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Abstract 

Portfolio  selection  problems  with  combinatorially-large  alternative  sets  can  be  impossible 
to  evaluate  precisely  on  a  reasonable  timescale.  When  portfolios  require  complex  modeling  for 
performance  assessment,  prohibitive  computational  processing  times  can  result.  Eliminating  a 
small  number  of  alternatives  through  an  intelligent  screening  process  can  greatly  reduce  the 
number  of  alternative  combinations,  thereby  decreasing  a  problem’s  evaluation  time  and  cost. 

A  methodology  was  developed  for  the  class  of  hierarchical  portfolio  selection  problems 
in  which  multiple  objectives  are  all  judged  on  the  same  sub-objectives.  First,  a  novel  capability- 
based  alternative  screening  process  was  devised  to  identify  and  remove  poor  alternatives,  thereby 
reducing  the  number  of  portfolios.  Then,  a  performance-based  portfolio  screening  process  was 
explored  to  estimate  portfolio  sufficiency  according  to  the  perfonnance  requirements  of  the 
decision  maker.  Following  the  establishment  of  a  set  of  sufficient  portfolios,  the  analyst  can 
employ  higher  resolution  post-analysis  methods  to  choose  a  final  solution. 

Finally,  the  methodology  was  applied  to  a  portfolio  selection  problem  in  which  the 
United  States  Strategic  Command  attempts  to  select  an  ideal  mix  of  intelligence,  surveillance, 
and  reconnaissance  assets.  After  deconstructing  the  actual  objective  hierarchy,  a  set  of 
representative  alternatives  were  evaluated  and  a  variety  of  screening  procedures  were  applied  to 
demonstrate  significant  reduction  in  the  number  of  possible  portfolios. 
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Definitions 


Goal 


The  overall  goal  of  a  decision  problem  defines  what  the  decision  maker  wishes  to 
accomplish.  In  an  evaluative  hierarchy,  the  goal  sits  alone  at  the  top  and  can  usually  be  assessed 
by  a  maximization  or  minimization  function. 


Objective 

The  set  of  objectives  determines  what  must  be  accomplished  for  the  decision  maker  to 
meet  the  goal.  Objectives  may  compete:  increased  achievement  in  one  objective  may  come  at 
the  cost  of  decreased  achievement  in  another  objective. 


Sub-objective 

Sub-objectives  separate  a  single  objective  into  its  component  parts,  to  better  understand 
how  well  an  objective  is  achieved. 


Measure 

A  measure  is  a  performance  indicator  on  which  direct  data  can  be  obtained.  A  set  of 
measures  is  used  to  determine  achievement  in  a  sub-objective.  In  the  literature,  the  term  ’metric' 
is  also  commonly  used  for  this  bottom  level  of  the  hierarchy. 
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SCREENING  AND  SUFFICIENCY  IN  MULTIOBJECTIVE  DECISION  PROBLEMS 

WITH  LARGE  ALTERNATIVE  SETS 


I.  Introduction 

Complex  decision  problems  require  careful  study  to  choose,  with  confidence,  an 
appropriate  solution  to  meet  the  overall  goal.  The  job  of  the  analyst  is  to  reduce  this  complexity 
to  a  model  that  reasonably  replicates  actuality  and  adds  value  to  the  decision  making  process. 
Difficulties  can  manifest  in  the  problem  structure,  with  competing  objectives,  or  in  the  solution 
pool,  with  many  alternatives  available.  When  the  solution  pool  is  large,  it  may  be  beneficial  to 
eliminate  poor  alternatives  to  allow  for  better  examination  of  the  reduced  set  of  alternatives. 

I.A.  Background 

Multicriteria  Decision  Analysis. 

The  methods  of  decision  analysis  (DA)  are  implemented  to  create  a  framework  to 
objectively  analyze  a  complex  decision  problem.  Input  from  the  decision  maker  and  subject 
matter  experts  is  used  to  evaluate  how  well  potential  solutions,  called  alternatives,  achieve  the 
overall  goal.  When  a  problem  features  several  potentially-competing  objectives,  multicriteria 
decision  analysis  (MCDA)  may  be  used.  The  purpose  of  MCDA  is  to  assist  the  decision  maker 
in  sorting  alternatives  into  groups,  ranking  alternatives  in  an  order,  or  selecting  a  best  alternative. 
At  a  fundamental  level,  MCDA  can  be  used  to  eliminate  alternatives  that  do  not  appear  to 
warrant  further  attention  (Hobbs  &  Meier,  2000).  Screening  the  alternative  set  creates  a  smaller 
universe  of  potential  solutions,  from  which  an  ideal  solution  may  be  more  easily  identified. 

While  multicriteria  decision  analysis  processes  are  helpful  for  many  problems, 
complicating  features  can  exist.  Because  MCDA  requires  each  alternative  to  be  evaluated 
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against  a  hierarchy  derived  from  the  decision  maker's  values  and  preferences,  the  analyst  must 
have  access  to  detailed  performance  data.  Determining  the  perfonnance  of  each  alternative  can 
prove  prohibitive  in  a  problem  with  a  very  large  number  of  alternatives.  A  screening  process 
may  be  helpful  to  reduce  the  number  of  alternatives  that  must  be  precisely  evaluated. 

Identifying  and  eliminating  dominated  alternatives  that  will  never  prove  ideal  can  reduce 
the  size  of  the  alternative  set.  The  resulting  screened  set  of  alternatives,  nearly  guaranteed  to 
contain  the  best  solution,  can  then  be  examined  more  closely.  When  the  problem  under  study  is 
to  develop  an  ideal  portfolio  comprised  of  several  alternatives,  screening  is  especially  important 
due  to  the  combinatorial  nature  of  the  number  of  potential  portfolios.  A  method  to  reduce  the 
alternative  set  to  a  more  tractable  size  is  vital. 

Combinatorial  Difficulties. 

Portfolio  selection  problems  featuring  alternative  sets  that  grow  combinatorially  present 
significant  difficulty  in  computation  and  evaluation.  Consider  an  alternative  set  that  contains 
elements.  A  -combination  of  set  is  a  subset  of  elements  from  .  Thus, 

The  subset  may  have  any  number  of  elements  from  one  to  .  The  number  of  possible  - 
combinations  of  is  represented  by 


(1) 


where  is  the  number  of  elements  in  and  ,  the  factorial,  is  defined  as  follows: 


(2) 
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When  the  solution  to  a  problem  is  a  portfolio  of  alternatives,  the  size  of  the  portfolio  is 
generally  unknown:  will  one  alternative  solve  the  problem,  are  three  needed,  are  ten  needed? 
Thus,  out  of  the  set  of  potential  alternatives  to  be  included  in  the  final  portfolio,  every  size 

portfolio  must  be  considered.  The  total  number  of  possible  portfolios  becomes 


(3) 


This  sum  grows  rapidly  with  each  new  alternative  introduced,  as  shown  in  Table  1. 


Table  1:  Combinatorial  growth 


Alternatives 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

15 

20 

30 

Combinations 

1 

3 

7 

15 

31 

63 

127 

255 

511 

1023 

32767 

1048575 

1073741823 

Thus,  if  one  can  eliminate  a  small  number  of  alternatives,  the  number  of  combinations  of  those 
alternatives  is  greatly  reduced,  as  seen  in  Figure  1 . 
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Combinatorial  Growth 


Figure  1:  Graph  of  combinatorial  growth 


Table  2  illustrates  the  dramatic  reduction  in  combinations  possible. 


Table  2:  Combination  reduction 


Size  of  n 

%  Reduction  in 

Alternatives 

%  Reduction  in 

Combinations 

10% 

50.05% 

10 

20% 

75.07% 

30% 

87.59% 

10% 

75.00% 

20 

20% 

93.75% 

30% 

98.44% 

10% 

87.50% 

30 

20% 

98.44% 

30% 

99.80% 

For  a  problem  with  ten  alternatives,  eliminating  one  alternative  (or  ten  percent)  reduces 
the  number  of  combinations  to  be  evaluated  by  over  half.  On  a  larger  problem  with  30 
alternatives,  eliminating  ten  percent,  just  three  of  the  alternatives,  reduces  the  number  of 
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combinations  by  nearly  88%;  eliminating  thirty  percent  of  the  alternatives  reduces  the  number  of 
combinations  by  99.8%.  As  the  size  of  the  alternative  pool  grows  even  larger,  eliminating  a  tiny 
fraction  of  alternatives  drastically  reduces  the  number  of  portfolios  that  must  be  considered. 

Screening. 

Screening  the  alternative  set  to  marginally  reduce  its  cardinality  can  greatly  reduce  the 
number  of  combinations  and  thereby  the  time  and  cost  associated  with  evaluation.  When 
portfolios  require  complex  modeling  for  performance  evaluation,  prohibitive  computational 
processing  times  can  result. 

Once  a  decision  is  made  to  screen  alternatives,  it  becomes  necessary  to  utilize  a  proper 
screening  methodology.  The  analyst  must  establish  appropriate  screening  factors  and  determine 
at  what  level  criteria  should  be  set  so  that  potentially  useful  alternatives  are  not  mistakenly 
eliminated.  An  alternative  must  provide  sufficient  performance  in  supporting  objectives  to 
effectively  achieve  the  goal.  Defining  this  sufficiency  level  allows  for  appropriate  screening  — 
any  solution  that  is  insufficient  can  be  screened  out.  This  sufficiency  process  applies  when  one 
alternative  must  be  selected  or  when  a  portfolio  of  alternatives  must  perform  sufficiently  to 
achieve  the  goal. 

I.B.  Problem  Statement 

Portfolio  selection  problems  with  a  large  number  of  alternatives  can  be  impossible  to 
precisely  evaluate  on  a  reasonable  timescale  due  to  both  data  collection  and  computation 
difficulties.  Eliminating  alternatives  through  an  intelligent  screening  process  can  greatly  reduce 
the  number  of  portfolios  under  consideration.  This  shortens  problem  evaluation  time  and  allows 
for  closer  examination  of  the  remaining  quality  portfolios. 
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In  this  paper,  a  method  is  developed  to  examine  a  specific  class  of  problems  with  very 
large  alternative  sets.  First,  a  novel  capability-based  alternative  screening  process  is  devised  and 
progressively  applied  to  several  screening  situations.  Then,  a  perfonnance-based  portfolio 
screening  process  is  explored  to  estimate  portfolio  sufficiency  according  to  the  perfonnance 
requirements  of  the  decision  maker.  Once  a  set  of  sufficient  portfolios  has  been  established,  the 
analyst  can  use  higher  resolution  post-analysis  methods  to  choose  a  final  solution. 

I.C.  Research  Scope 

The  class  of  hierarchical  problems  under  study  consists  of  those  in  which  an  overall  goal 
is  met  through  achievement  in  multiple  objectives;  all  objectives  are  judged  on  the  same  set  of 
sub-objectives.  However,  measures  of  performance  for  the  sub-objectives  are  allowed  to  vary 
among  objectives.  The  decision  maker  wishes  to  select  an  ideal  portfolio  of  alternatives  best  to 
meet  the  overall  goal. 

The  methodology  is  most  useful  when  a  large  number  of  alternatives  produces  an 
extremely  large  set  of  possible  portfolios,  making  evaluation  of  each  portfolio's  perfonnance 
impossible.  This  research  develops  a  two-step  process  to  reduce  the  number  of  alternatives  for 
problems  of  this  common  structure  and  then  determine  which  remaining  portfolios  sufficiently 
meet  the  decision  maker's  requirement. 

I.D.  Assumptions 

The  methodology  assumes  a  problem  subject  to  its  application  can  be  structured  as 
previously  described.  Additionally,  the  performance  of  alternatives  is  assumed  to  be  determined 
through  evaluation  against  a  hierarchical  model  that  has  been  rigorously  created  through  finn 
DA  methodologies. 
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I.E.  Thesis  Organization 

The  remainder  of  this  thesis  contains  four  chapters:  Chapter  II  explores  literature  in  areas 
pertinent  to  solving  problems  of  this  structure;  Chapter  III  synthesizes  concepts  from  the 
literature  review  to  develop  a  methodology  for  problem  analysis;  Chapter  IV  progressively 
applies  the  developed  methodology  to  a  current  problem  and  analyzes  the  results;  finally, 
Chapter  V  discusses  relevant  conclusions  and  examines  opportunities  for  further  research. 
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II.  Literature  Review 


II.A  Introduction 

The  purpose  of  this  literature  review  is  to  explore  thoroughly  the  areas  of  sufficiency  and 
alternative  screening  in  multicriteria  decision  analysis.  A  brief  description  of  decision  analysis 
will  be  provided,  followed  by  longer  explorations  of  the  aforementioned  fields.  As  these  areas 
are  less  developed  in  the  literature,  a  comprehensive  examination  will  be  made  in  an  effort  to 
fully  understand  which  elements  should  be  considered  for  future  methodology  creation. 

II.B  Decision  Analysis 

The  field  of  decision  analysis  (DA),  a  subset  of  the  broader  field  of  Operations  Research, 
"provides  effective  methods  for  organizing  a  complex  problem  into  a  structure  that  can  be 
analyzed"  (Clemen  &  Reilly,  2001).  Generally,  a  decision  maker  must  select  an  alternative  that 
will  best  solve  the  problem  faced.  A  best  practice  to  assist  the  decision  maker  is  to  create  a 
hierarchical  model  of  objectives  and  values  according  to  the  philosophy  of  value  focused 
thinking  (VFT).  VFT  approaches  a  decision  problem  by  first  identifying  what  needs  to  be 
achieved  —  the  objectives.  These  objectives  are  each  broken  down  into  what  is  desirable  in  an 
ideal  solution  —  the  values  (Clemen  &  Reilly,  2001).  These  values  are  reduced  to  a  set  of 
measureable  metrics;  potential  solutions  are  generated  and  then  evaluated  against  these  metrics. 

VFT  is  preferred  to  alternative  focused  thinking,  which  simply  examines  the  pre-existing 
set  of  alternatives  and  selects  the  best  among  them,  thus  eliminating  the  opportunity  to  identify 
and  explore  alternatives  not  initially  considered. 

Several  desirable  properties  exist  for  the  value  hierarchy  developed.  The  hierarchy 
should  exhibit  completeness,  meaning  the  evaluation  considerations  at  each  tier  cover  all 


concerns  necessary  for  overall  objective  evaluation.  The  hierarchy  must  be  nonredundant,  so 
that  no  two  evaluation  considerations  in  the  same  tier  overlap.  The  hierarchy  must  be 
decomposable,  meaning  each  evaluation  criteria  is  independent  of  others.  The  hierarchy  should 
be  operable,  so  it  is  easily  understood  by  anyone  who  will  use  it.  Finally,  it  is  desirable  to  have  a 
hierarchy  of  small  size  while  still  encapsulating  all  necessary  elements  (Kirkwood,  1997). 

Ideally,  the  hierarchical  model  will  provide  a  consistent  systematic  framework  for  alternative 
evaluation  and  decision  justification  (Dawley,  Marentette,  &  Long,  2008). 

II.C  Sufficiency 

Many  problems  ask  the  fundamental  question:  "how  much  is  enough?"  When  evaluating 
alternatives  to  achieve  a  non-binary  objective  (i.e.,  there  is  not  an  obvious  success/fail  point),  the 
decision  maker  generally  acknowledges  a  certain  minimum  level  of  accomplishment  that  is 
acceptable.  Whatever  alternative  is  chosen,  it  must  meet  this  level  to  be  considered  a  sufficient 
solution.  Beyond  this  level,  a  "better"  alternative  may  provide  more  value  toward  meeting  the 
objective.  However,  any  additional  resource  requirements  of  this  "better"  alternative  may  be  too 
high  to  justify  the  marginal  increase  in  value  achievement. 

The  decision  maker  wishes  to  maximize  perfonnance  on  an  objective  or  set  of  objectives 
while  minimizing  the  resources  necessary  for  that  achievement.  While  this  fundamental  question 
of  sufficiency  is  approached  in  decision  problems  in  diverse  fields,  little  effort  has  been  made  to 
formally  investigate  how  to  establish  sufficiency. 

Military  sufficiency. 

The  military  deals  frequently  with  questions  of  sufficiency  in  regards  to  long-tenn  force 
structure.  Because  of  the  lengthy  defense  acquisition  cycle,  it  is  vital  to  project  what  may  be 
needed  under  a  variety  of  circumstances.  Since  the  Cold  War,  sufficiency  has  been  a  key  part  of 
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the  discussion  on  deterrence.  The  United  States  must  have  "sufficient  military  capabilities 
available  to  provide  a  wide  range  of  flexible  options  for  military  and  non-military  conflict" 
(DuBridge,  1969).  This  sufficient  force  must  be  able  to 

deter  an  attack  upon  the  United  States  or  its  allies;  respond  at  a  level  appropriate 
to  situations  short  of  an  all-out  attack  on  the  United  States;  retaliate  effectively, 
after  an  all-out  attack  on  the  United  States;  and,  ensure  national  survival  in  the 
event  of  an  attack.  (Temple  III,  2005) 

In  the  Cold  War  environment  with  memories  of  Pearl  Harbor  still  fresh,  defense  sufficiency 
meant  maintaining  a  force  structure  with  the  ability  to  respond  to  any  and  all  provocations. 

More  recently,  the  United  States  Navy  studied  sufficiency  in  its  surface  combatant  force 
structure  for  a  25-year  horizon  (Morris,  2000).  The  Navy  recognized  that  traditional  wargame 
modeling  techniques  based  on  global  war  theory  are  insufficiently  broad  in  scope  and  incapable 
of  high  fidelity  modeling,  and  thus  inappropriate  to  model  local  warfare  and  non-military 
operations.  A  future  force  structure  needs  to  perform  peacetime  missions  efficiently  but  also 
manage  a  quick  transition  to  wartime  readiness  if  necessary. 

Traditional  Cold  War-era  battle  groups  are  inefficient  for  joint  and  asymmetric  warfare  as 
standardized  groups  possess  either  too  little  or  too  much  capability.  The  operating  area  of  a 
surface  combatant  was  modeled  through  location  of  the  attack  target  and  the  reach  of  available 
weapons.  Large  operating  areas  enable  a  single  surface  combatant  to  contribute  to  multiple 
tasks,  which  tends  to  reduce  the  number  of  ships  needed  to  perfonn  the  whole  set  of  tasks.  For 
example,  strategic  land-attack  missions  could  be  performed  with  Tomahawk  missiles  from 
almost  any  ship  in  the  theatre,  but  for  all  other  tasks  the  assigned  ships  would  have  to  be  near  the 
asset  being  defended  or  attacked  (Morris,  2000). 
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The  Johns  Hopkins  Applied  Physics  Laboratory  developed  a  'sufficiency  analysis' 
methodology  to  determine  the  types  and  numbers  of  assets  needed  to  perform  a  task.  The 
process  decomposed  a  complex  multi-area  warfare  problem  into  a  set  of  single-area  problems, 
solved  these  simpler  problems,  then  integrated  the  results  to  obtain  a  solution.  Success  criteria 
detennined  whether  a  set  of  surface  combatants  was  able  to  accomplish  a  task  at  an  acceptable 
level  of  risk.  These  criteria  were  defined  by  quantifiable  measures  of  effectiveness  (MOE)  with 
threshold  values  assigned  for  each  measure.  The  risk  for  a  task  was  deemed  acceptable  if  and 
only  if  all  MOEs  exceeded  the  thresholds;  typical  MOE  thresholds  were  0.9  in  terms  of 
probability  of  success. 

One  method  used  to  set  these  thresholds  was  to  find  the  point  of  diminishing  return  —  the 
"knee"  on  the  curve  of  effectiveness  versus  number  of  assets  dedicated  to  a  task.  This  did  not 
account  for  failure  consequence  severity  or  attack  likelihood,  so  a  Naval  subject  matter  expert 
used  professional  judgment  to  detennine  the  amount  of  risk  considered  acceptable.  The  study 
concluded  that  a  surface  force  mixture  should  consist  of  fewer,  highly-capable  ships  as  opposed 
to  a  larger  number  of  less-capable  ships,  as  overall  force  size  could  be  minimized  while 
capability  could  be  maximized  when  a  preponderance  of  the  force  consisted  of  the  very  best 
ships.  The  concept  of  sufficiency  of  force  was  developed  through  combining  known  capabilities 
with  subject  matter  expert  opinions  on  risk  and  future  needs. 

General  sufficiency  theory. 

Advances  in  technology  and  changes  in  the  international  political  landscape  have 
transformed  the  role  of  the  analyst.  Frequently,  a  decision  must  be  made  when  it  is  unknown 
whether  there  are  enough  pieces  of  infonnation  or  if  the  infonnation  available  is  of  high  enough 
quality  to  make  a  good  decision.  The  decision  maker  is  subject  to  the  "supervisor's  dilemma", 
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defined  as  "a  generic  situation  wherein  a  supervisor  must  decide  if  the  output  product  of  an 
analysis  is  acceptably  rigorous  or  if  more  analytical  resources  must  be  invested  in  that  analysis 
process  before  sending  it  forward"  (Zelik,  Woods,  &  Patterson,  2009).  The  decision  maker 
wishes  to  know  if  the  data  available  and  analysis  performed  is  sufficient  to  move  ahead  with  the 
decision  making  process. 

Zelik  et  al.  develop  an  approach  to  determine  analysis  sufficiency  based  on  rigor  of 
process  as  an  indicator  of  infonnation  sufficiency.  "Given  the  current  analytical  production 
pressures  and  the  technology  driven  proliferation  of  data  availability  shaping  the  intelligence 
community,  it's  increasingly  difficult  to  accurately  judge  the  sufficiency  of  an  analysis"  (Zelik, 
Patterson,  &  Woods,  2007).  To  adapt  to  these  changes,  improvement  in  the  understanding  of  the 
analytical  process  was  investigated.  Analysis  sufficiency  is  judged  on  the  basis  of  rigor  in  the 
analytical  process.  "Rigor,  as  an  assessment  of  process  quality,  is  used  in  information  analysis  to 
communicate  about  the  process,  rather  than  the  product,  of  analysis"  (Zelik,  Patterson,  &  Woods, 
2007). 

It  is  difficult  to  know  when  sufficient  data  analysis  has  been  performed.  By  adhering  to 
an  accepted  and  rigorous  analytical  process,  one  can  feel  confident  in  the  level  of  analysis. 
Achieving  'maximum'  rigor  of  process  is  not  a  goal;  rather,  one  should  continually  adapt  the 
process  of  analysis  to  changes  in  the  world  to  ensure  the  analysis  is  rigorous  enough.  The 
supervisor's  dilemma  captures  the  fundamental  tradeoffs  inherent  in  analysis  work,  by  balancing 
urgency  and  limited  resources  against  the  need  to  perform  broad  research  to  gain  insight. 

By  presenting  a  subject  pool  of  analysts  and  decision  makers  to  the  structure  of  an 
analysis  (not  the  actual  analysis  itself),  Zelik  et  al.  attempted  to  gain  an  understanding  of  how 
process  influences  quality.  The  data  indicate  that  providing  insight  into  an  analysis  process 
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influences  assessments  of  both  process  rigor  and  product  quality.  Participants  made  insightful 
comments  about  the  quality  of  an  analytic  process  based  on  product  quality,  but  these 
perceptions  were  apt  to  change  with  the  addition  of  process  insight.  This  distinguishes  perceived 
rigor,  based  on  cues  inferred  from  an  analytic  product,  from  effective  rigor,  based  on  insight  into 
the  analytic  process;  these  two  areas  may  not  always  be  aligned.  Subject  responses  demonstrated 
there  is  no  one  right  way  to  perform  analysis.  Different  practices  can  produce  acceptable  results, 
and  approaches  can  be  valid  even  if  not  completely  understood. 

This  examination  of  process  as  a  measuring  stick  for  analysis  sufficiency  helps  prevent 
overconfidence  in  inadequate  analysis  (and  likewise  underconfidence  in  strong  analysis).  Such 
overconfidence  in  superficially-adept  analysis  can  lead  to  catastrophic  results.  The  investigation 
into  the  loss  of  the  space  shuttle  Columbia  showed  that  critical  decisions  were  made  based  on 
analyses  that  appeared  to  be  thorough  but  was  in  reality  of  low  rigor.  Using  the  strength  of 
analysis  as  a  proxy  for  product  quality  advances  the  true  goal  of  the  analytic  effort  —  not 
providing  individual  facts  or  reports,  but  providing  information  and  understanding  in  a 
meaningful  context  (Zelik,  Patterson,  &  Woods,  2007). 

The  supervisor's  dilemma  does  not  advocate  'maximum'  rigor,  it  concerns  determining 
what  is  sufficient  rigor  when  balancing  in  the  costs  of  further  work.  In  this  sense,  dedicating 
further  resources  to  a  problem  —  resources  that  could  be  allocated  elsewhere  —  can  actually 
decrease  the  value  of  that  research.  The  theory  of  "just  barely  good  enough"  (JBGE)  in  agile 
modeling  and  computer  programming  suggests  that  time  and  effort  spent  above  and  beyond  the 
minimum  requirement  adds  little  extra  functionality  to  the  solution  and  decreases  the  overall 
value  of  the  product  as  the  cost  to  capability  ratio  increases.  The  JBGE  model,  by  including  the 
cost  of  a  solution  in  the  overall  understanding  of  "value"  obtained,  presents  the  most  efficient 
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solution  possible.  It  does  not  imply  low  quality,  as  the  threshold  for  sufficiency  is  set  at  what  is 
acceptable.  The  point  when  JBGE  is  reached  is  situational  and  dynamic  as  needs  change,  and 
generally  comes  sooner  than  expected  (Ambler  S.  ,  2002). 

If  a  model  or  solution  must  meet  a  pre-determined  requirement,  then  once  it  hits  that 
sufficiency  level  —  once  it  is  good  enough  for  effective  use  —  then  anything  beyond  that  is  a 
waste  of  time  or  money.  The  point  of  diminishing  returns  has  been  reached;  further  capability 
might  be  added,  but  it  is  not  necessary  and  comes  at  a  higher  cost  than  minimum  development, 
thereby  decreasing  overall  value.  The  graph  of  value  of  performance  when  cost  is  considered  as 
a  function  of  effort,  shown  in  Figure  2,  is  not  a  monotonically  increasing  function  but  a  parabola 
with  the  local  maxima  representing  the  JBGE  point.  This  point  achieves  the  highest  possible 
value  as  a  function  of  cost,  as  any  less  effort  produces  insufficient  capability,  while  any  more 
effort  produces  more,  unneeded  capability,  at  a  higher  cost  that  diminishes  overall  return. 


Figure  2:  Just  barely  good  enough  (Ambler  S.  W.,  2009) 
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Three  key  tenants  must  be  followed  with  respect  to  this  theory.  First,  JBGE  is  situational, 
in  that  what  defines  'good  enough’  changes  based  on  the  ultimate  needs  both  now  and  in  the 
future.  The  point  of  JBGE  may  shift  as  demands  change.  Re-working  a  product  to  meet  a  less 
complex  requirement  is  wasted  time,  unless  the  current  version  decreases  value  elsewhere,  as 
may  be  the  case  when  the  model  is  used  to  determine  data  analysis  needs. 

Second,  any  requirement  must  be  clearly  agreed  upon  by  the  analyst  and  the  user/decision 
maker,  so  that  the  product  is  good  enough  to  function  and  that  effort  is  not  wasted  on 
functionality  that  will  not  be  used.  Finally,  and  most  importantly,  the  point  of  JBGE  comes 
quicker  than  expected,  as  the  value  in  modeling  "comes  through  improved  communication  and 
ability  to  think  things  through,  and  that  this  value  peaks  rather  quickly"  (Ambler  S.  W.,  2009). 
Human  nature  is  to  err  on  the  conservative  side,  but  this  must  be  avoided  as  not  to  waste 
resources.  Development  should  aim  for  sufficiency. 

Sufficiency  in  fields  of  study. 

Though  the  literature  contains  limited  investigation  into  the  theory  of  sufficiency  in  data 
analysis,  concepts  of  sufficiency  in  application  to  a  specific  field  are  discussed  more 
prominently.  Much  research  has  been  devoted  to  the  concept  of  a  sufficient  statistic  —  a  statistic 
which  has  the  property  of  sufficiency  with  respect  to  a  statistical  model  and  its  associated 
unknown  parameter,  such  that  "no  other  statistic  which  can  be  calculated  from  the  same  sample 
provides  any  additional  information  as  to  the  value  of  the  parameter  to  be  estimated"  (Fisher, 
1922).  Most  commonly  used  in  regards  to  discovering  an  underlying  distribution,  a  statistic  is 
sufficient  if  the  sample  from  which  it  is  calculated  gives  no  additional  information  than  does  the 
statistic. 
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An  example  of  this  application  is  detennining  how  much  data  is  necessary  to  be  confident 
in  the  distribution  of  a  population  of  unknown  size.  Veres  et  al.  studied  sufficiency  in  biometric 
data  collection,  estimating  how  many  subjects  are  needed  to  cover  diversity  of  population  in  gait 
and  how  many  samples  per  subject  are  needed  to  provide  representation  of  the  similarities  and 
differences  in  the  gait  of  a  single  subject.  Their  investigation  into  when  the  data  is  sufficient  to 
have  statistically  significant  results  for  chosen  values  of  Type  1  and  Type  2  error  reinforces  the 
concept  of  diminishing  returns  on  data  collection.  After  some  population  size,  the  number  of 
subjects  needed  to  characterize  that  population  stays  practically  the  same,  and  there  is  almost  no 
value  in  expending  resources  to  collect  further  data  (Veres,  Nixon,  &  Carter,  2006). 

A  similar  question  of  how  much  data  is  sufficient  to  determine  an  underlying  trend  is 
found  in  the  analysis  of  physician  performance.  Measuring  the  true  quality  of  physician 
perfonnance  is  important  to  a  range  of  healthcare  initiatives.  Most  tools  use  a  performance  ratio 
based  on  the  number  of  patients  with  the  diagnosis  of  interest,  with  a  minimum  number  of 
patients  needed  to  establish  a  valid  physician  score.  For  a  single  perfonnance  measure  with  a 
high  level  of  unreliability,  as  many  as  100  patients  with  a  particular  condition  might  be  necessary 
to  measure  a  physician's  perfonnance  (Lumetra,  2005). 

However,  the  amount  of  data  needed  can  be  reduced  by  using  a  composite  score, 
combining  performance  results  across  multiple  single  indicators  of  perfonnance.  Such 
aggregated  measures  reduce  the  unreliability  in  the  measurement  and  thus  the  number  of  patients 
with  a  particular  medical  condition  needed.  As  few  as  25  patients  per  doctor  may  be  enough  to 
compute  perfonnance  measures  that  accurately  discriminate  perfonnance  among  providers.  The 
key  is  to  use  the  right  mix  of  data  sources,  which  is  unique  to  each  problem.  Combining 
different  types  of  data,  each  with  a  different  reliability  level,  can  provide  a  clearer  picture  than  a 
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larger  amount  of  a  single  type  of  data.  It  may  be  possible  to  more  quickly  gain  insight  into  the 
characteristics  of  the  subject  of  study  through  the  use  of  less,  but  aggregated,  data. 

The  use  of  less  data  than  is  expected  to  be  necessary  is  important  to  resource 
conservation.  Monitoring  traffic  system  intersections  to  determine  real-time  flow  distributions 
requires  significant  resource  investment,  as  "control  systems  are  often  hungry  for  extensive 
detector  information  from  a  rich  detector  configuration"  (Lan,  2001).  Lower  levels  of 
information  may  be  sufficient  for  the  control  system  to  perfonn  its  task.  Reducing  the  collectors 
needed  to  determine  system  behavior  in  one  area  allows  for  additional  infonnation  to  be  gathered 
elsewhere,  increasing  overall  system  knowledge. 

Lan  investigated  how  partial  sets  of  detectors  provide  less,  yet  still  sufficient, 
information,  albeit  with  the  penalty  of  less  accuracy  and  greater  variability.  In  a  simple  three- 
way  intersection  example,  a  configuration  with  five  detectors  instead  of  six  can  still  provide 
necessary  and  sufficient  data  for  system  control.  More  generally,  testing  on  the  Jacobian  matrix 
of  the  predicted  parameter  outputs  of  traffic  count  at  an  intersection  can  determine  whether  a 
detector  configuration  can  provide  sufficient  information  for  flow  estimation.  With  this  method, 
"as  long  as  one  can  generate  an  a  priori  reasonable  set  of  input  counts,  no  actual  data  are  needed 
to  perfonn  [the  analysis]"  (Lan,  2001).  The  capabilities  of  the  system  can  be  examined  based 
solely  on  its  own  properties,  without  the  need  for  exact  perfonnance  data. 

In  the  traffic  example,  a  set  of  detectors  must  continually  monitor  the  changing  status  of 
the  system.  The  field  of  change  point  analysis  provides  techniques  to  determine  when  an  abrupt 
change  in  state  occurs.  Discovering  when  a  party  of  interest  makes  a  change  is  vital  to 
intelligence  gathering.  For  example,  an  enemy  base  may  store  a  certain  number  of  tanks  —  this  is 
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the  default  state.  When  the  tanks  are  no  longer  at  the  base,  the  state  has  changed,  and  the  analyst 
can  then  try  to  detennine  where  the  tanks  went  and  what  this  may  mean  in  the  larger  scheme. 

In  such  situations,  change  point  theory  may  be  applied.  Commonly  used  in  industrial 
quality  control,  change  point  detection  (also  known  as  statistical  process  control)  is  used  to 
continuously  monitor  a  process  and  determine  when  an  underlying  distribution  has  changed. 
Consider  a  manufacturing  plant  manager  who  is  interested  in  detecting  when  the  quality  of  the 
product  has  decreased.  While  a  single  observation  can  be  used  (i.e.,  a  certain  measurement  on  a 
widget  must  be  within  a  tolerance  to  meet  the  specification),  it  is  also  possible  to  observe  a 
process  through  multiple  data  streams.  This  multivariate  surveillance  is  useful  if  any  number  of 
potential  changes  to  different  parts  of  a  system  can  indicate  a  change  point.  A  surveillance 
procedure  should  have  a  high  probability  of  detecting  a  change  within  a  reasonable  period  of 
time,  with  a  low  probability  of  false  alarm  (Wessman,  1998). 

If  dense  data  streams  are  monitored  automatically,  an  alarm  procedure  must  be  developed 
to  identify  the  change  point.  This  can  be  done  through  a  univariate  process  based  on  a 
summarizing  statistic,  or  analysis  of  the  separately-monitored  component  processes.  Wessman 
explored  a  likelihood  ratio-based  surveillance  procedure  in  processes  where  a  sudden  shift  occurs 
between  two  fully-specified  alternatives  that  is  reflected  in  all  component  processes.  In  a 
multivariate  manufacturing  example,  the  physical  breakdown  of  a  machine  or  machine  part 
would  cause  a  number  of  observable  data  distributions  to  suddenly  change.  In  an  intelligence 
situation,  a  large  shift  in  enemy  equipment  would  result  in  a  change  to  the  "distribution"  of  the 
expected  observation. 

Wessman  examined  sufficient  reductions  to  change  detection  methods  that  do  not  suffer 
from  significant  loss  of  efficiency.  For  processes  where  the  change  is  between  two  fully 
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specified  alternatives,  when  the  change  points  from  multiple  intelligence  streams  occur 
simultaneously,  a  univariate  likelihood  ratio  statistic  method  is  sufficient  for  detecting  certain 
critical  events.  Correspondingly,  a  good  surveillance  procedure  can  utilize  this  univariate 
process. 

Interest  in  continual  statistical  surveillance  with  regards  to  intelligence  gathering  has 
increased  in  recent  years,  with  the  aim  of  detennining  when  an  important  change  in  an 
underlying  process  has  occurred  (Frisen,  Andersson,  &  Schioler,  2009).  This  type  of 
surveillance  involves  collecting  data  from  several  related  variables,  calling  for  multivariate 
surveillance  techniques.  Optimality  in  terms  of  data  collection  is  hard  to  derive  and  even  hard  to 
define  in  such  multivariate  problems.  It  is  possible  that  multivariate  surveillance  problems  can 
be  simplified  by  the  sufficiency  principle,  which  states  that  all  conclusions  to  be  drawn  should 
depend  only  on  a  sufficient  statistic. 

In  intelligence  surveillance  there  is  the  potential  for  complex  relations  between  change 
points,  ranging  from  simultaneous  changes  to  independent  changes.  One  method  for 
simplification  is  to  reduce  the  variate  vector  into  one  statistic  and  then  use  a  system  for 
univariate  surveillance  on  this  statistic.  Monitoring  each  variable  separately,  in  a  combined 
univariate  method,  is  also  possible.  A  common  method  to  combine  the  information  is  to  signal 
the  alann  at  the  first  time  that  any  of  the  univariate  methods  gives  an  alann  (Frisen,  Andersson, 
&  Schioler,  2009).  In  a  situation  where  only  one  process  changes,  the  performance  is 
considerably  improved  if  this  knowledge  is  utilized  in  the  surveillance  procedure. 

II.D  Alternative  Screening 

Once  sufficiency  requirements  for  a  set  of  alternatives  have  been  detennined,  the  set 
should  be  screened  to  isolate  quality  alternatives.  Screening  criteria  are  used,  each  of  which 
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"consists  of  an  attribute  and  a  cutoff  level  which  divide  areas  [alternatives]  into  those  which  are 
acceptable  and  those  which  are  not"  (Keeney,  1980). 

Belton  and  Stewart  (2003)  state  that  "sometimes  the  problem  is  not  one  of  generating 
alternatives,  but  one  of  identifying  an  appropriate  and  manageable  set  for  detailed  evaluation 
from  a  much  larger  set  of  possibilities  —  a  screening  process."  The  set  should  be  made  up  of 
"good"  alternatives,  or  alternatives  which  represent  the  range  of  possibilities.  Strategies  to 
accomplish  this  include  bounding  the  space  of  promising  alternatives,  or  using  a  simplified 
assessment  model. 

Chen  (2006)  observes  that  "during  the  past  few  decades  several  different  methods  have 
been  separately  put  forward  to  deal  with  screening  problems.  But  there  has  been  no  systematic 
exploration  of  this  topic  in  the  literature,  and  researchers  on  sorting  [ordering  solutions]  have 
paid  little  attention  to  it."  Similar  to  sufficiency  theory  in  data  analysis,  there  has  been  little 
advanced  investigation  into  how  screening  criteria  should  be  established.  Instead,  general 
guidelines  for  screening  advocate  caution. 

Performance  screening. 

Various  options  for  screening  on  alternative  performance  are  available.  The  following  is 
a  partial  list  developed  by  the  National  Institute  of  Standards  and  Technology.  For  each  of  these 
screening  methods,  the  threshold  levels  are  set  by  subject  matter  experts. 

Dominance 

An  alternative  is  dominated  if  another  alternative  out-performs  it  with  respect  to 
at  least  one  attribute,  and  performs  [at  least]  equally  with  respect  to  the  remainder 
of  attributes.  With  the  Dominance  method,  alternatives  are  screened  such  that  all 
dominated  alternatives  are  discarded.  The  screening  power  of  this  method  tends  to 
decrease  as  the  number  of  independent  attributes  becomes  larger. 

Conjunctive  (“Satisficing  ”) 

The  Conjunctive  method  is  purely  a  screening  method.  The  requirement 
embodied  by  the  Conjunctive  screening  approach  is  that  in  order  to  be  acceptable, 
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an  alternative  must  exceed  given  performance  thresholds  for  all  attributes.  The 
attributes  (and  thus  the  thresholds)  need  not  be  measured  in  commensurate  units. 

Disjunctive 

The  Disjunctive  method  is  also  purely  a  screening  method.  It  is  the  complement 
of  the  Conjunctive  method,  substituting  “or”  in  place  of  “and.”  That  is,  to  pass  the 
Disjunctive  screening  test,  an  alternative  must  exceed  the  given  perfonnance 
threshold  for  at  least  one  attribute.  Like  the  Conjunctive  method,  the  Disjunctive 
method  does  not  require  attributes  to  be  measured  in  commensurate  units. 

Lexicographic 

Using  this  method,  attributes  are  rank-ordered  in  terms  of  importance.  The 
alternative  with  the  best  performance  on  the  most  important  attribute  is  chosen.  If 
there  are  ties  with  respect  to  this  attribute,  the  next  most  important  attribute  is 
considered,  and  so  on.  (Norris  &  Marshall,  1995) 

Given  the  lack  of  direction  in  screening  theory,  many  multicriteria  decision  problems 
utilize  rudimentary  screening  techniques  to  conservatively  eliminate  undesirable  alternatives. 

In  the  field  of  mineral  extraction,  a  company  must  make  a  binary  decision  whether  to  drill 
for  ore.  This  decision  is  based  on  data  indicating  if  the  lode  is  large  enough  and  of  sufficient 
quality  to  invest  significant  resources.  Venna  (2001)  investigated  the  application  of  fuzzy  logic 
in  mineral  resource  evaluation.  Screening  criteria  were  established  for  mineral  quality  measures 
of  perfonnance,  then  instead  of  crisp  cutoffs,  a  fuzzy  logic  process  for  combining  measures  was 
used.  Measure  sensitivity  analysis  allowed  for  more  rigorous  detennination  of  the  sufficiency  of 
the  deposit  for  drilling. 

The  United  States  Forest  Service  (1999),  while  developing  a  monitoring  program  for  the 
environmental  health  of  forests  in  the  Northwest,  suggested  the  following  criteria  for  measures 
used  for  screening: 

-  the  dynamics  of  the  measure  parallel  the  larger  system 

-  the  measure  shows  a  short-tenn  but  persistent  response  to  change  in  the  status  of  the 
system 
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-  measure  can  be  accurately  and  precisely  estimated  (high  signal-to-noise  ratio) 

-  likelihood  of  detecting  a  change  in  the  measure  performance  is  high,  given  a  true  system 
change 

-  low  natural  variability,  and  changes  in  measure  values  can  be  distinguished  from 
background  variation 

-  cost  of  measure  obtainment  is  not  prohibitive 

While  not  all  of  these  characteristics  will  apply  to  each  screening  problem,  the  Forest  Service 
emphasizes  that  measures  should  be  good  reflections  of  the  alternatives  they  are  measuring, 
especially  when  used  for  screening. 

Feasibility  screening. 

The  previous  section  discussed  screening  methods  based  on  alternative  performance. 
Another  option  is  feasibility  screening  —  the  United  States  Department  of  the  Interior  Bureau  of 
Reclamation  advocates  only  eliminating  alternatives  that  are  impossible,  as  anything  that  is 
possible  must  have  its  effort  of  implementation  evaluated  as  part  of  its  scoring.  If  a  possible 
alternative  is  eliminated  because  it  seems  too  difficult  prior  to  a  full  evaluation,  a  solution  may 
be  missed.  Only  options  with  "fatal  flaws"  should  be  eliminated  (Reclamation's  Decision 
Process  Guide). 

If  an  alternative  (or  set  of  alternatives)  is  capable  of  providing  value,  then  it  should  not  be 
eliminated.  Parnell  et  al.  state: 

Screening  criteria  should  avoid  targeting  feasible  but  less  desirable  alternatives. 
Feasibility  screening  sifts  out  alternatives  based  on  non-negotiable  criteria.  For 
all  others,  finding  the  best  trade-offs  will  lead  to  the  perfect  solution. . . .  All 
feasible  alternatives  become  solution  candidates.  Balancing  trade-offs  to  find  the 
preferred  solution  is  accomplished  by  enhancing  and  measuring  solution 
candidates.  (2008) 
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Screening  criteria  should  not  be  so  strict  that  sufficient  but  superficially  undesirable 
alternatives  are  eliminated.  Detennining  which  alternatives  are  undesirable  is  the  task  of 
perfonnance  screening.  By  eliminating  alternatives  prematurely,  the  power  of  the  process  is 
reduced. 

This  leads  to  the  question  of  how  "non-negotiable"  is  defined,  and  how  such  criteria 

should  be  established.  These  absolute  criteria  are  established  by  real-world  limitations  for  the 

problem  at  hand  and  the  firm  demands  of  the  subject  matter  experts  consulting  on  the  problem. 

Parnell  suggests  screening  should  be  done  at  the  ’needs'  level,  not  at  the  'wants'  or  'desires'  level. 

Screening  criteria  should  be  used  to  ensure  solutions  meet  minimum  requirements.  These 

criteria  can  then  be  consulted  as  alternatives  are  designed,  modeled  and  analyzed.  Belton  and 

Stewart  state  that  any  such  process  should: 

screen  alternatives  in  order  to  exclude  ones  which  are  'non-complianf  in  that  they 
do  not  meet  certain  minimum  specifications.  This  should  always  be  done  with 
care,  making  sure  that  a  degree  of  non-compliance  on  one  criterion  could  not  be 
compensated  for  by  exceptional  performance  elsewhere.  (2003) 

ft  is  warned  not  to  screen  on  individual  measures  when  overall  alternative  perfonnance  can  be 

improved  by  success  in  other  measures.  Belton  and  Stewart  advocate  using  a  subject  matter 

expert  panel  to  create  screening  criteria  to  reduce  the  alternatives  to  a  smaller  set,  of  which  the 

ideal  solution  probably  exists. 

Keeney  (1980)  advocates  using  a  two-step  process,  in  which  absolute  requirements 
dictate  screening  criteria.  Once  alternatives  have  been  screened  for  feasibility,  a  set  of 
evaluation  criteria  are  developed.  Each  alternative  is  then  scored  against  the  evaluation  criteria 
for  sorting.  The  initial  screening  criteria,  generated  by  real-world  limitations,  eliminate  options. 
The  evaluation  criteria,  developed  by  subject  matter  experts,  are  used  to  sort  alternatives  relative 
to  each  other. 
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Definitions. 

Chen  et  al.  (2008)  performed  research  on  screening  in  multiple  criteria  decision  analysis, 
screening  alternatives  in  multiple  criteria  subset  selection,  and  distance  methods  for  multiple 
criteria  decision  aid.  Their  work  provides  techniques  to  reduce  the  number  of  alternatives  in 
multiple  criteria  subset  selection  problems,  thereby  making  it  less  difficult  to  find  a  good 
portfolio.  The  main  objective  of  screening  is  to  remove  inferior  alternatives  from  the  set  of 
available  alternatives,  so  that  those  remaining  can  be  investigated  in  more  detail,  using  more 
accurate  data  or  more  refined  assessment  criteria. 

When  utilizing  preference  expressions  on  values  and  weights  to  fonn  a  decision  support 
system,  resulting  alternatives  may  be  similar.  Caution  should  be  used,  as  screening  should 
provide  the  decision  maker  a  range  of  options  that  emphasize  different  attributes.  The  analyst 
should  be  careful  not  to  "stack  the  deck"  in  favor  of  a  particular  set  of  values,  nor  yield  a  set  of 
options  that  are  essentially  similar  (Chen,  Kilgour,  &  Hipel,  2008). 

In  a  MCDA  problem  there  exist  a  number  of  alternatives  ,  the  complete  set  of 
which  is  identified  by 


(4) 

Alternatives  are  evaluated  against  each  metric,  or  measure,  ;  the  group  of  measures 
comprises 


(5) 
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The  performance  of  an  alternative  for  a  measure 


is  represented  by 


,  as  shown  in 


Table  3. 


The  decision  maker  may  value  increasing  an  alternative's  measure  perfonnance  until  a 
point  is  reached  when  little  additional  value  is  created.  A  low  level  of  perfonnance  in  a  measure 
may  have  very  little  value  to  the  decision  maker;  value  would  be  achieved  only  when  a  rough 
threshold  is  cleared.  The  true  value  of  an  alternative's  perfonnance  in  a  measure  is  not  always  a 
linear  function,  and  is  expressed  as: 


(6) 

where  the  value  of  the  performance  is 
the  decision  maker.  The  value  vector  of  an 
represented  by: 


determined  by  the  value  function  established  by 
alternative's  perfonnance  in  the  set  of  measures  is 


(7) 
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Different  measures  may  have  different  levels  of  importance  to  the  decision  maker.  Thus, 
it  is  necessary  to  assign  a  global  weight  to  each  measure  .  Weights  are  non-negative 
and  sum  to  one: 


(8) 

(9) 


with  the  weight  vector 

In  the  basic  case,  once  the  measure  perfonnance  and  values  have  been  evaluated,  the 
aggregate  value  score  for  an  alternative,  ,  is  calculated: 


(10) 

Once  the  value  scores  for  each  alternative  are  derived,  the  decision  making  can  begin: 
choosing  the  best  alternative,  ranking  the  alternatives,  or  sorting  the  alternatives  into  groups 
(Chen,  Kilgour,  &  Hipel,  2008). 

Screening,  the  next  analytical  step,  selects  a  non-empty  subset  of  (defined  as  ) 
that  hopefully  contains  the  ideal  solution, 

(ID 

(12) 


26 


The  set  of  alternatives  eliminated  by  a  screening  procedure  is  represented  by 


(13) 


The  process  of  screening  is  represented  pictorially  in  Figure  3. 


More 
preferred 
group  ^ " 


* 


Figure  3:  Screening  schematic  (Chen,  Kilgour,  &  Hipel,  2008) 

For  a  sorting  screening  procedure,  when  alternatives  are  divided  into  two  (or  more) 
preference  groups,  only  preferences  between  groups  can  be  made,  not  between  alternatives 
within  the  same  group. 


(14) 

(15) 
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where  is  defined  as  is  preferred  over  ,  and  is  defined  as  is 

neither  more  or  less  preferred  than 

Screening  eliminates  poor  alternatives  to  allow  further  analysis  to  focus  on  quality 
alternatives.  A  screening  process  should  possess  three  characteristics:  safety,  efficiency,  and 
good  infonnation. 

A  screening  procedure  is  safe  iff : 


(16) 


A  safe  screening  procedure  ensures  the  ideal  choice  remains  in  the  screened  set. 

Given  two  safe  screening  procedures  and  ,  is  more  efficient  than  if 
.  The  more  infonnation  the  decision  maker  provides,  the  more  alternatives  can  be 
eliminated.  Thus,  screening  procedure  must  be  based  on  better  infonnation  than  screening 
procedure 

When  an  initial  screen  is  insufficient  for  the  decision  maker,  Chen  et  al.  (2008)  advocate 
the  sequential  use  of  various  screening  methods  to  arrive  at  a  subset  of  alternatives  for  further 
examination. 

Kilgour  et  al.(2004)  explore  subset  selection  knapsack  problems.  Cunent  screening 
techniques  are  poorly  suited  for  subset  selection  problems  as  they  can  eliminate  choices  that, 
when  combined  with  other  alternatives,  would  produce  the  best  possible  subset.  In  a  resource- 
constrained  knapsack  problem,  not  all  individually  dominated  alternatives  can  be  safely  removed 
from  consideration.  However,  several  conditions  exist  in  which  a  dominated  alternative  cannot 
possibly  belong  to  an  optimal  subset.  Without  constraints,  the  problem  simply  becomes  an  m- 
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best  alternative  problem.  Finally,  sensitivity  analysis  on  weightings  should  be  conducted  so 
alternatives  are  not  eliminated  that  belong  to  an  ideal  subset  under  slightly  different  conditions. 

Dominance  can  be  used  for  screening.  If  an  alternative  is  worse  than  another  alternative 
in  all  performance  or  capability  measures,  then  depending  on  the  problem  type  the  dominated 
alternative  can  be  discarded  (Kilgour,  Rajabi,  Hipel,  &  Chen,  2004). 


(17) 


Distance  method. 

Chen  et  al.  developed  a  case-based  distance  method  to  screen  alternatives.  Obtaining 
accurate  decision  maker  preference  on  weights  and  values  can  be  difficult,  so  this  method 
discards  that  process.  Instead,  a  test  set  of  alternatives  is  presented  to  the  decision  maker  who 
chooses  which  subset  of  the  test  set  is  acceptable.  This  test  set  may  include  real  past  decisions, 
fictitious  but  realistic  alternatives,  and  a  representative  subset  of  familiar  alternatives. 

A  test  set  must  contain  alternatives  that  mirror  all  aspects  of  the  real  alternatives 
under  consideration,  including  identical  measures.  The  distance  method  proposed  by  Chen  et  al. 
(2008)  suggests  that  the  acceptable  alternatives  should  be  'close  together',  while  the 
unacceptable  alternatives  should  be  'outside'  such  that: 

(18) 

(19) 
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Thus, 


(20) 

(21) 

The  test  set  is  used  to  estimate  the  measure  weights  ,  as  well  as  a  distance  threshold 
The  normalized  distance  of  the  alternatives  in  from  the  ideal  (and  imaginary) 
center  of  is  less  than  ,  while  the  nonnalized  distance  of  the  alternatives  in  is  greater 
than 

Using  the  weighting  developed  from  this  test  set,  a  distance-based  screening  procedure  is 
used  to  evaluate  the  real  alternatives  in  .  Alternatives  within  a  certain  distance  of  the 
imaginary  ideal  alternative  ,  the  centroid  of  ,  are  selected  for  further  analysis.  With 
and  ,  the  measure  performance  of  the  ideal  alternative  is  determined  by: 


(22) 


With  the  properties  of  calculated,  a  weighted  squared  Euclidian  method  provides 
distance  information.  For  all  ,  the  nonnalization  factor  is  defined  as: 


(23) 
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Thus,  the  distance  between  and  for 


is: 


(24) 


and  the  distance  between  and  for  is 


(25) 


The  total  distance  between  and  is 


(26) 


Here,  is  calculated  using  an  optimization  procedure  (Chen, 

Kilgour,  &  Hipel,  2008).  However,  if  weights  are  known  from  the  decision  maker,  these  can  be 
used  instead.  Similarly,  if  the  decision  maker  can  outline  a  true  perfect  alternative,  its  measure 
perfonnance  can  be  used  in  lieu  of  the  centroid  of  .  Applying  the  distance  procedure  to  the 

set  of  real  alternatives,  the  total  distance  between  and  is 


(27) 
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and 


(28) 


The  distance  threshold  is  also  calculated  from  the  optimization  procedure,  and  is 
derived  from  ,  the  set  of  acceptable  test  alternatives.  Alternatively,  can  be  adjusted  to 
by  the  decision  maker  to  reduce  or  expand  to  an  acceptable  level.  The  screened  set  is 
defined  by: 


(29) 


When  using  a  distance  method,  screening  procedure  is  more  efficient  than  screening 
procedure  if  the  subset  of  alternatives  produced  by  is  contained  within  the  subset 
produced  by  .  The  screening  procedure  that  produces  the  smallest  safe  subset  is  deemed 
efficient.  This  distance  based  screening  procedure  is  illustrated  in  Figure  4. 
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43. 


Figure  4:  Case  based  distance  method  screening  procedure  (Chen,  Kilgour,  &  Hipel,  2008) 


II.E  Summary 

This  chapter  presented  first  a  brief  review  of  the  concepts  of  decision  analysis  and  value 
focused  thinking.  Further  discussion  on  these  areas  can  be  found  in  the  cited  materials.  A 
deeper  exploration  was  then  made  of  sufficiency  and  alternative  screening. 

The  literature  strongly  suggests  sufficiency  to  be  defined  by  those  familiar  with  the 
problem,  including  the  decision  maker  and  subject  matter  experts.  Defining  the  level  of 
perfonnance  sufficiency  is  important,  as  any  effort  beyond  the  requirement  may  be  a  waste  of 
resources.  Several  studies  have  found  sufficiency  may  be  reached  with  less  data  than  expected. 
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Regarding  military  sufficiency,  it  is  important  to  account  for  all  possible  scenarios  to  ensure 
flexibility  and  readiness. 

Previous  research  advises  caution  when  screening  and  sorting.  Performance  screening 
should  be  conservative,  as  not  to  eliminate  potentially-good  alternatives  if  conditions  change. 
Many  advocate  eliminating  only  impossible  solutions,  defined  by  laws  of  nature  or  strict  decision 
maker  policy. 

The  next  chapter  combines  the  screening  and  sufficiency  concepts  discussed  in  the 
literature  with  Chen's  distance-based  procedure  to  develop  a  methodology  for  alternative 
screening  and  sufficiency  evaluation  in  hierarchical  problems  with  large  alternative  sets. 
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III.  Methodology 


III.A.  Introduction 

This  chapter  extends  the  definition  of  the  problem  type  under  consideration  and  develops 
a  capability-based  method  for  initial  alternative  screening.  Additionally,  a  method  is  described 
to  establish  portfolio  sufficiency  levels,  allowing  for  performance-based  portfolio  screening. 

III.B  Problem  Description 

Complex  decision  problems  can  demand  achievement  in  a  large  number  of  objectives  to 
satisfy  an  overall  goal.  In  many  situations,  these  objectives,  which  may  be  competing,  are 
judged  on  the  same  core  criteria.  Within  each  objective  the  criteria,  or  sub-objectives,  may  have 
varying  importance.  A  potential  solution  is  then  evaluated  on  these  sub-objectives  to  detennine 
how  well  it  meets  the  objectives  and  thus  the  goal.  When  a  large  alternative  set  exists,  the 
problem  can  easily  grow  too  large  to  precisely  evaluate  all  alternatives  or  combinations  of 
alternatives,  either  from  a  data  collection  or  computation  standpoint.  In  situations  such  as  this,  it 
is  beneficial  to  trim  the  number  of  alternatives  to  a  more  manageable  set.  This  smaller  set  of 
quality  alternatives  can  then  be  examined  more  closely. 

Consider  a  company  that  wishes  to  achieve  internal  control  over  its  transportation  needs. 
The  chosen  strategy  is  to  procure  a  fleet  of  vehicles  that  fulfills  a  variety  of  needs:  long  haul 
transportation  of  goods,  local  transportation  of  goods,  executive  transportation,  etc.  Each  of 
these  need  domains  represents  an  objective  for  the  company;  an  appropriate  solution  will  procure 
enough  vehicles  of  different  types  to  meet  all  objectives  and  thus  meet  the  overall  goal  of 
transportation  control.  Notably,  these  objectives  compete:  a  vehicle  that  is  effective  at  long  haul 
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transportation  of  goods  is  poor  at  executive  transportation.  Two  of  these  objectives,  for 
simplicity  given  equal  importance,  are  shown  in  Figure  5. 


Overall  Goal 


Objectivesweight 


Figure  5:  Company  trasportation  objectives 


Any  vehicle  could  theoretically  achieve  some  level  of  value  in  each  objective  (even  if 
that  value  is  very  close  to  zero).  As  such,  each  objective  is  evaluated  on  the  same  set  of  vehicle 
capability  sub -objectives.  A  sample  of  sub-objectives  used  to  determine  objective  achievement 
may  be:  highway  performance  (HP),  goods  capacity  (Cap),  and  passenger  comfort  (Comf). 
However,  as  objective  requirements  vary  widely,  the  sub-objectives  under  each  objective  are 
weighted  differently,  as  seen  in  the  superscripts.  Figure  6  expands  on  the  Figure  5  hierarchy  by 
adding  three  evaluative  sub-objectives  under  each  objective. 
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Figure  6:  Identical  sub-objectives  define  each  objective 
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Achievement  in  a  sub-objective  is  determined  by  value  obtained  in  set  of  associated 
performance  measures.  Data  is  gathered  directly  on  the  performance  measures.  In  this  example, 
highway  performance  is  measured  through  cost  per  mile  traveled  (CPM).  Capacity  is  measured 
in  two  parts:  goods  capacity  and  people  capacity.  Comfort  is  measured  through  the  amenities 
available  to  a  passenger.  Measures  may  be  categorical;  with  comfort,  increasing  value  is 
assigned  to  escalating  luxury  level.  The  updated  performance  hierarchy  is  shown  in  Figure  7. 
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Figure  7:  Performance  measures  determine  sub-objective  value  achievement 

The  vehicles  available  for  purchase  are  the  alternatives.  The  company  can  select  a 
portfolio  of  vehicles  that  best  meets  its  transportation  needs  by  evaluating  each  possible  portfolio 
against  the  performance  hierarchy.  As  there  are  hundreds  of  vehicles  on  the  market  to  choose 
from,  there  exists  a  very  large  number  of  possible  portfolios.  Eliminating  some  of  these  vehicles 
from  consideration  will  reduce  the  number  of  possible  portfolios  and  allow  for  more  manageable 
evaluation. 
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III.C  Hierarchy  Consolidation 

Background. 

This  problem  is  currently  structured  with  a  performance-based  hierarchy.  The  standard 
evaluation  method  is  to  gather  precise  performance  data  across  all  measures  for  all  alternatives. 

If  the  number  of  alternatives  is  very  large,  this  data  may  be  difficult  or  even  impossible  to  obtain. 
When  the  solution  involves  selecting  a  portfolio,  each  of  which  requires  evaluation  against  the 
hierarchy,  the  combinatorial  nature  of  evaluating  all  possible  alternative  combinations  becomes 
prohibitive.  To  use  the  hierarchy  at  a  later  time,  alternative  performance  data  must  be  updated, 
requiring  significant  recollection  cost. 

To  combat  this  difficulty,  the  problem  structure  can  be  transformed  from  a  performance- 
based  hierarchy  to  a  capability-based  hierarchy.  As  the  sub-objectives  are  the  same  for  each 
objective,  the  sub-objective  local  weights  can  be  transfonned  into  global  weights  and  summed, 
thus  determining  which  sub-objectives  are  most  important  to  the  overall  goal.  Then,  the  large 
multi-objective  performance-based  hierarchy  is  reduced  to  a  capability-based  hierarchy  featuring 
only  sub-objectives.  To  detennine  their  potential  contribution  to  the  overall  goal,  alternatives 
can  be  rated  on  a  binary  capability  scale  of  whether  they  contribute  to  a  sub-objective,  and  true 
performance  data  (which  may  be  difficult  or  impossible  to  gather)  can  be  ignored.  Sub-objective 
value  functions  or  even  exact  measures  of  performance  are  no  longer  necessary,  greatly 
eliminating  data  collection  costs.  In  this  way,  alternatives  that  offer  high  capability  can  be 
identified  as  likely  to  be  included  in  a  good  portfolio. 

Approach. 

A  decision  problem  can  be  structured  in  three  different  ways.  The  decision  maker  can  be 
tasked  with  i)  selecting  the  best  alternative  from  the  set  ,  ii)  sorting  into  different  groups 
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that  are  arranged  in  a  preference  order,  and  iii)  ranking  the  alternatives  of  from  best  to  worst 
(Chen,  Kilgour,  Hipel  2004). 

The  procedure  presented  here  focuses  on  the  second  structure,  specifically  sorting  the 
alternatives  in  into  two  groups  —  one  selected  for  further  examination  and  one  to  be 
eliminated  .  This  will  be  accomplished  by  evaluating  alternatives  on  the  new  capability- 
based  hierarchy.  This  approach  can  be  used  on  problems  where  the  sub-objectives  are  identical 
for  a  set  of  objectives  and  alternative  capability  (if  an  alternative  is  designed  to  exhibit  a  specific 
capability)  is  understood. 

True  alternative  performance  for  a  capability  is  not  necessary,  nor  is  a  value  function 
relating  that  perfonnance  into  sub-objective  value  achievement.  In  this  procedure,  sub-objective 
and  measure  weights  are  known  via  subject  matter  experts  and/or  the  decision  maker,  as  is  the 
perfonnance  of  an  imaginary  ideal  alternative.  Both  of  these  pieces  of  information  are  generally 
easily  obtained;  if  necessary,  this  information  can  be  obtained  indirectly  using  the  method  of 
Chen  et  ah,  a  linear  programming  optimization  procedure  using  alternative  test  sets. 

Procedure. 

A  multi-criteria  decision  analysis  problem  is  explored  in  which  a  series  of  top  objectives 
are  all  evaluated  on  the  same  set  of  sub-objectives  .  Each  objective  is  distinct,  yet  the 
underlying  hierarchy  is  repeated,  as  shown  in  Figure  8.  The  perfonnance  measures  may  be 
evaluated  differently  among  the  sub-objectives,  as  appropriate  and  needed. 
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Figure  8:  Full  performance-based  hierarchy 

While  the  sub-objectives  are  the  same  for  each  objective,  they  may  possess  different 
importance  depending  on  the  objective,  and  thus  are  assigned  appropriate  local  weights.  The 
objectives  can  have  different  importance  to  the  overall  goal,  but  for  simplicity  here  they  are  all  of 
equal  weight.  This  is  a  small  performance-based  hierarchy  with  only  two  objectives;  a  complex 
multiobjective  decision  analysis  problem  may  have  over  ten  objectives,  generating  dozens  of 
sub-objectives  and  potentially  hundreds  of  measures.  If  perhaps  20  alternatives  are  under 
consideration,  data  for  thousands  of  performance  measures  must  be  gathered.  This  would 
require  significant  resource  allocation  for  the  initial  evaluation;  any  future  change  in 
performance  measures  or  alternative  capability  would  require  additional  data  collection.  Most 
importantly,  evaluating  every  combination  of  the  20  alternatives  against  the  entire  hierarchy 
would  demand  significant  time  and  resources. 

It  is  necessary  to  reduce  the  performance-based  hierarchy  to  a  capability-based  hierarchy. 
To  do  this,  the  measures  are  ignored  while  the  sub-objectives  are  featured.  The  sub-objective 
local  weights  are  transformed  to  global  weights,  as  shown  in  Figure  9. 
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Figure  9:  Elimination  of  performance  measures 

In  the  example  shown  in  Figure  9,  ,  and  ,  the  number  of  distinct 

sub-objectives.  The  goal  of  this  transformation  is  to  determine  which  of  the  sub¬ 

objectives  are  most  important.  It  follows  that  alternatives  that  provide  that  capability  in  the 
important  sub-objectives  are  highly  desirable.  Here,  is  the  total  global  weight  of  ,  while 
is  the  global  weight  of  .  There  are  objectives. 


(30) 

(31) 

The  vector  provides  the  relative  importance  of  the  sub-objectives  towards  meeting  the  entire 
set  of  objects.  This  creates  the  capability-based  hierarchy  seen  in  Figure  10. 
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Figure  10:  Capability-based  hierarchy 


Now,  an  alternative  is  evaluated  directly  on  the  sub-objectives  —  specifically,  if  it  is 
designed  to  provide  a  capability.  If  yes,  then  the  alternative  scores  a  "1"  for  that  sub-objective;  if 
not,  the  alternative  scores  a  "0."  Instead  of  a  number  of  precise  performance  measure  data  points 
for  each  alternative,  only  knowledge  of  whether  that  alternative  is  designed  to  achieve  capability 
in  a  sub-objective  is  needed.  Evaluating  each  alternative  against  the  reduced  hierarchy  allows 
for  to  be  screened  into  and  ,  with  size  of  each  according  to  the  rigor  required  by  the 
decision  maker. 

III.D  Alternative  Screening 

Once  a  large  performance-based  hierarchy  is  converted  to  a  capability-based  hierarchy,  a 
modified  distance  method  can  be  used  to  screen  the  alternative  set.  As  previously  described, 

,  the  set  of  alternatives  considered.  Alternatives  are  evaluated  against  each 
sub-objective  in  the  reduced  capability-based  hierarchy.  The  group  of  sub-objectives 
comprises 

In  this  methodology,  an  alternative  is  evaluated  simply  on  whether  it  is  designed  to 
provide  a  capability  in  a  sub-objective.  Measure  performance  and  resulting  measure  value 
become  obsolete.  Instead,  an  alternative's  performance  in  a  sub-objective,  defined  as  , 
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is  based  purely  on  capability,  and  becomes  a  binary  0  or  1  response.  Thus,  the  value  an 
alternative  achieves  for  that  capability  in  a  sub-objective,  defined  as  ,  is  also  a  binary 
response.  The  set  of  values  for  the  sub-objective  capabilities  of  alternative  is  represented  by 

,  a  string  of  Os  and  Is. 

As  different  sub-objectives  may  have  different  levels  of  importance  to  the  decision 
maker,  the  reduction  procedure  changes  local  weights  (assigned  by  subject  matter  experts)  to 
global  weights  for  each  sub-objective  ,  with  and  the  weight  vector 

Once  alternatives  are  scored  against  the  capability-based  hierarchy,  various  screening 
procedures  can  be  assessed.  In  a  basic  approach,  one  could  screen  based  on  total  value  to  keep  a 
certain  number  of  alternatives  that  provide  the  highest  independent  capability.  Or,  one  can 
remove  dominated  alternatives.  For  example,  if  an  alternative  provides  capability  in  a  certain  set 
of  sub-objectives  but  there  exists  another  alternative  that  provides  those  same  capabilities  plus 
more,  the  first  alternative  is  dominated  and  would  be  removed.  Care  must  be  taken  when 
applying  such  aggressive  screening  without  considering  performance,  and  should  only  be  applied 
to  specific  problems  when  capability  is  truly  binary  with  little  perfonnance  variation. 

Finally,  one  could  evaluate  portfolios  of  alternatives  in  a  knapsack-type  procedure  to 
ensure  no  capability  gaps  exist  (a  danger  when  screening  based  on  highest  independent 
capability).  This  procedure  can  be  implemented  if  additional  constraints  exist  —  number  of 
potential  alternatives,  cost  of  alternatives,  etc.  —  and  can  be  structured  to  the  specific  problem 
through  linear  programming  techniques. 

A  modified  distance-based  screening  procedure  is  used  to  evaluate  the  real  alternatives  in 
.  The  subject  matter  experts  have  provided  the  local  sub-objective  weights  (which  the  analyst 
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has  changed  to  global  weights)  and  the  ideal  alternative  is  known  —  that  which  provides 
capability  in  every  sub-objective.  Alternatives  within  a  certain  distance  of  are  selected  for 
further  analysis. 

With  the  properties  of  the  ideal  known, 


(32) 

the  weighted  squared  Euclidian  method  previously  presented  provides  distance  information.  For 
all  ,  the  nonnalization  factor  is  now  reduced  to: 


(33) 


Thus,  the  distance  between  and  for  is 


(34) 


and  the  distance  between  and  for  is 


(35) 
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So,  the  total  distance  between  and  is 


(36) 


and  =  {  ,  where  is  set  by  the  decision  maker  to  select  a  certain  screened 

sample  size.  Increasing  increases  the  number  of  alternatives  in 

Each  alternative  is  examined  for  dominance.  Starting  with  ,  is  computed,  then 
is  compared  to  ;  if  contains  's  in  the  same  vector  positions  as  another 

,  and  that  contains  a  in  an  additional  position,  then  is  dominated  and  removed  from 
.  This  is  repeated  for  .  Again,  caution  should  be  used  when  implementing  this 

rudimentary  screening  procedure  to  ensure  helpful  alternatives  are  not  eliminated. 

Alternatively,  dominance  can  be  used  to  screen  by  eliminating  alternatives  that  only 
provide  strength  in  the  sub-objectives  with  lowest  global  weight.  If  an  alternative  only  provides 
functionality  to  an  area  of  low  concern,  it  will  not  be  among  the  highest-rated  alternatives  and 
will  probably  not  appear  in  a  highly-ranked  portfolio.  These  simple  methods  of  removing 
alternatives  to  reduce  the  model  size  can  help  when  later  running  performance-based  models 
with  lengthy  computing  times.  As  previously  stressed,  truly  eliminating  alternatives  risks 
accidentally  removing  a  piece  of  an  ideal  portfolio;  screening  should  be  perfonned  carefully  and 
depend  on  the  specific  problem  under  evaluation.  Screening  methods  to  differentiate  "good"  and 
"poor"  alternatives  can  be  used  to  compile  a  quality  portfolio  to  begin  a  search  algorithm; 
starting  with  a  solution  closer  to  the  ideal  will  lower  search  times. 
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III.E  Sufficiency  Evaluation 

The  methodology  developed  thus  far  can  be  used  when  the  process  of  gathering  accurate 
perfonnance  data  on  a  large  set  of  alternatives  or  alternative  combinations  is  costly  or  even 
impossible.  It  is  used  to  screen  alternatives  to  reduce  the  size  of  the  alternative  set.  Next,  the 
analyst  must  evaluate  the  remaining  possible  portfolios,  ensuring  the  portfolio  alternatives  are 
sufficient  to  satisfactorily  solve  the  problem.  The  analyst  should  set  the  distance  threshold  to 
reject  truly  insufficient  solutions  while  allowing  sufficiently  helpful  solutions  to  remain.  In  a 
problem  with  limited  data  this  may  be  difficult;  there  is  little  guidance  in  the  literature  on  how  to 
determine  which  solutions  may  be  sufficient.  Previous  work  leaves  this  problem  element  to  the 
decision  maker  or  subject  matter  experts. 

Combining  input  from  the  top-level  decision  maker  and  bottom-level  analysts,  a 
minimum  sufficiency  level  for  portfolios  of  alternatives  can  be  established.  This  level  can  be 
utilized  in  the  secondary  perfonnance-based  portfolio  screening  procedure.  In  many  situations, 
the  overall  minimum  acceptable  performance  for  a  solution  is  generally  know,  or  at  least  the 
desired  performance.  In  a  problem  with  significant  uncertainty,  the  difficulty  is  detennining 
which  alternative  sets  are  likely  to  achieve  that  perfonnance. 

First,  the  analysts  who  generate  perfonnance  measure  data  establish  value  functions  for 
their  specific  area  of  expertise.  These  value  functions,  based  on  past  experience,  feature 
increasing  levels  of  perfonnance  on  the  x-axis  with  the  value  derived  from  that  level  of 
performance  on  the  y-axis,  creating  a  monotonically-increasing  function.  While  a  perfonnance- 
based  hierarchy  may  have  a  large  number  of  performance  measures,  each  value  function  is 
created  only  once  —  a  reasonable  task  given  a  large  number  of  analysts  with  specific  knowledge 
of  each  perfonnance  measure. 
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Next,  the  top-level  decision  maker  is  asked  what  level  of  certainty  he  wishes  with  this 
analysis.  For  example,  is  the  overall  decision  extremely  important  and  requires  high  certainty 
that  the  screened  alternative  set  will  work  —  say,  90 %?  Or  perhaps  a  larger,  less-certain  set  of 
alternatives  would  be  okay  —  a  confidence  level  of  70%  may  suffice.  The  selection  of  a  certainty 
level  is  the  second  step. 

Third,  this  certainty  level  is  transferred  back  down  to  the  analyst  level.  The  confidence 
percentage  the  decision  maker  selects  is  found  on  the  y-axis  of  each  performance  measure  value 
function  graph,  and  the  value  function  itself  maps  this  level  to  the  data  analysis  requirement  on 
the  x-axis,  as  shown  in  Figure  1 1 .  Now,  the  decision  maker's  single  input  has  been  pushed 
through  the  hierarchy  to  determine  the  data  needs  to  arrive  at  that  level  of  certainty. 
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Figure  11:  Value  function 


Finally,  the  alternatives  can  be  screened  on  whether  they  meet  the  certainty  needs  of  the 
decision  maker.  If  an  alternative  (or  portfolio)  can  provide  the  data  at  the  levels  needed  by  the 
analysts  to  meet  the  decision  maker's  confidence  level,  then  that  alternative  is  selected  for  further 
study.  If  an  alternative  (or  portfolio)  cannot  provide  the  data  at  the  levels  needed,  then  that 
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alternative  is  screened  out.  Post-analysis  is  performed  on  the  sufficient  portfolios  examining  cost 
and  other  considerations.  Thus,  a  screening  procedure  based  on  data  sufficiency  is  used  to 
reduce  the  size  of  the  portfolio  set. 

III.F  Summary 

This  chapter  has  clarified  the  type  of  problem  under  consideration:  one  in  which 
achievement  in  a  series  of  distinct  domains  is  evaluated  through  performance  in  the  same  set  of 
criteria.  Additionally,  a  large  number  of  alternatives  exist  leading  to  a  combinatorially-large 
number  of  possible  portfolios  to  be  evaluated  against  the  hierarchy.  A  methodology  to  reduce 
the  number  of  alternatives  was  developed  with  the  goal  of  greatly  reducing  the  number  of 
portfolios  considered. 

A  process  of  transforming  a  perfonnance-based  hierarchy  to  a  capability-based  hierarchy 
was  described.  By  evaluating  individual  alternatives  on  capability,  reduction  in  the  size  of  the 
alternative  set  through  the  elimination  of  dominated  alternatives  is  achieved.  Then,  a  portfolio 
sufficiency  screening  process  was  developed  based  on  decision  maker  desires  and  analyst  needs. 
The  final  result  is  a  small  set  of  portfolios  ready  for  post-analysis  and  selection. 
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IV.  Results  and  Analysis 


IV.A  Introduction 

This  chapter  applies  the  developed  methodology  to  a  United  States  Department  of 
Defense  Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  platfonn  portfolio  selection 
problem.  The  problem  features  a  large  number  of  alternatives  leading  to  a  very  large  number  of 
possible  portfolios.  The  results  of  various  screening  procedures  are  discussed. 

IV.B  Background 

The  United  States  military  conducts  complex  operations  with  many  types  of  assets. 
Campaign  planners  traditionally  use  combat  modeling  techniques  to  detennine  the  most  efficient 
set  of  weapon  assets  necessary  to  achieve  victory.  It  is  preferred  to  use  the  minimum  set  of 
assets  that  will  provide  sufficient  capability  to  ensure  success. 

The  recent  proliferation  of  advanced  information  gathering  assets  has  coincided  with  the 
expansion  of  intelligence  requirements  stemming  from  the  wars  in  Iraq  and  Afghanistan.  More 
ISR  assets  are  available  to  commanders  than  ever  before.  However,  the  rapid  adaptation  of  these 
new  assets  during  wartime  has  led  to  inefficient  use.  Study  of  ideal  asset  distribution  has  been 
largely  ignored  as  military  and  civilian  leadership  has  demanded  immediate  deployment  in 
support  of  soldiers,  airmen,  and  marines. 

This  has  created  difficulties  in  efficiently  providing  data  to  the  hundreds  of  units  that 
require  infonnation  on  a  timely  basis.  Some  organizations  are  drowning  in  data  as  numerous 
assets  stream  terabytes  of  data  to  be  processed.  Meanwhile,  other  groups  are  suffering  from  a 
lack  of  the  data  needed  to  develop  actionable  intelligence.  ISR  assets  are  not  well  coordinated 
across  the  Department  of  Defense  to  ensure  an  appropriate  level  of  information  reaches  each 
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user.  Additionally,  the  patchwork  structure  of  ISR  asset  usage  across  services  risks  that  the 
assets  in  operation  may  not  be  working  in  an  efficient  hannony  to  maximize  data  obtainment. 

An  analysis  of  ISR  capability  is  needed  to  ensure  an  ideal  portfolio  of  ISR  assets  is  deployed  in 
today's  joint  environment  and  developed  for  future  requirements. 

Adding  to  the  difficulty  of  organizing  an  ideal  portfolio  is  the  fundamental  ambiguity  on 
the  effectiveness  of  a  deployed  ISR  asset.  The  clear  objectives  of  typical  air  assets  provide  a 
division  between  mission  initiation  and  mission  success.  With  strike  aircraft,  placing  a  bomb  on 
a  specific  target  provides  a  clear  definition  of  mission  success.  With  air  refueling  tankers, 
loitering  in  a  specific  airspace  during  a  specific  time  while  transferring  fuel  and  then  returning  to 
base  defines  a  successful  mission.  For  ISR  assets,  there  is  no  clear  measure  of  success  for  a 
specific  mission  —  the  aim  is  continual  data  gathering.  This  data  may  later  be  transfonned  into 
actionable  information  to  further  advance  the  war  effort,  but  there  is  not  necessarily  the 
immediate  recognition  of  mission  success  or  completeness  for  an  ISR  asset  during  a  specific 
sortie. 

Thus,  it  remains  difficult  to  model  ISR  tasking  appropriately.  With  traditional  strike 
aircraft,  capability  and  expected  effectiveness  are  known  traits  —  an  F-22  can  release  a  specific 
weapon  that  will  inflict  a  known  level  of  damage  with  a  given  level  of  certainty.  With  ISR 
assets,  the  amount  of  pure  data  needed  to  gain  a  certain  amount  of  actionable  intelligence  is 
variable  and  generally  unknown.  Thus,  unlike  traditional  force  modeling  where  platfonn 
capabilities  are  summed  until  the  necessary  force  to  meet  an  objective  is  created,  ISR  modeling 
requires  examining  the  amount,  quality,  and  type  of  information  needed  and  how  best  to  meet 
this  requirement. 
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IV.C  Application 

Hierarchy. 

The  United  States  Strategic  Command  (USSTRATCOM),  one  of  the  ten  Unified 
Combatant  Commands  administered  by  the  United  States  Department  of  Defense,  is  responsible 
for  intelligence,  surveillance,  and  reconnaissance  (ISR)  operations  for  the  United  States  military. 
USSTRATCOM  has  command  authority  over  the  Joint  Functional  Component  Command  for 
Intelligence,  Surveillance  and  Reconnaissance  (JFCC-ISR),  which  coordinates  global 
intelligence  collection  for  worldwide  Department  of  Defense  operations  and  national  intelligence 
requirements  (USSTRATCOM,  2010).  As  the  center  for  planning,  execution  and  assessment  of 
these  areas,  JFCC-ISR  has  been  tasked  to  develop  an  ISR  force  sizing  model.  The  organization 
has  used  multiobjective  decision  analysis  tools  to  create  a  hierarchical  model  to  measure  the 
quality  of  force  capabilities.  The  goal  is  to  develop  an  ISR  portfolio  that  optimizes  value 
achievement. 

The  hierarchy  consists  of  seven  top  level  objectives  which  define  domains  of  ISR 
necessity,  as  shown  in  . 


Figure  12:  JFCC-ISR  hierarchy 


Some  objectives  are  more  important  than  others  and  thus  are  weighted  appropriately; 
weights  are  indicated  below  the  objectives.  Each  objective  is  defined  by  an  identical  set  of  three 
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sub-objectives;  the  relative  weight  distribution  among  sub-objectives  varies  among  the 
objectives.  Performance  for  each  of  the  three  sub-objectives  is  detennined  by  three  identical 
measures;  the  weight  distribution  among  measures  varies  among  the  sub-objectives.  The 
hierarchy  for  the  third  objective  (of  seven)  is  presented  in  Figure  13. 
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Figure  13:  Objective  3  hierarchy 


This  top  level  Objective  3,  with  a  weight  of  0.448,  is  evaluated  through  three  sub¬ 
objectives  which  are  necessary  for  its  achievement:  INTI,  INT2,  and  INT3.  These  INT  sub¬ 
objectives  represent  specific  methods  of  intelligence  (INT)  collection.  The  weight  for  each  INT 
sub-objective,  representing  the  relative  importance  to  the  objective  achievement,  is  included.  All 
seven  objectives  are  evaluated  on  these  three  sub-objectives,  as  each  objective  requires  varying 
levels  of  the  three  types  of  intelligence  data  to  gain  understanding  in  the  domain.  As  certain 
types  of  intelligence  are  more  important  to  some  objectives,  the  weight  distribution  among  the 
sub-objectives  differs  for  each  objective. 

Finally,  each  INT  sub-objective  is  evaluated  on  the  same  three  measures:  quality, 
timeliness,  and  persistence.  Raw  data  from  an  ISR  asset  can  be  obtained  for  these  measures, 
which  are  rolled  up  to  evaluate  sub-objective  achievement.  The  weight  for  a  measure  represents 
its  relative  importance  to  achieving  the  higher-level  sub-objective;  weight  distribution  among  the 
three  measures  differs  among  sub-objectives  and  across  objectives. 
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The  hierarchy  presented  here  for  Objective  3  is  repeated  for  each  of  the  other  six 
objectives.  Success  in  each  objective  domain  is  achieved  through  the  same  means  — 
achievement  in  the  INTI,  INT2,  and  INT3  sub-objectives,  which  are  evaluated  through  the  same 
three  measures.  Thus,  the  seven  objectives  lead  to  21  sub-objectives  and  63  measures.  The  full 
hierarchy  is  shown  in  Appendix  1 . 

Alternatives. 

The  Department  of  Defense  deploys  a  number  of  assets  to  obtain  necessary  ISR  data. 
Through  the  development  of  a  clear  statement  of  priorities  and  the  derivative  hierarchy,  an  ideal 
set  of  assets  can  be  developed  to  meet  these  priorities.  This  ideal  portfolio  may  consist  of  any 
combination  of  available  ISR  assets.  For  some  assets,  multiple  equipment  packages  can  be 
installed,  or  the  asset  may  be  flown  from  various  locations  and  follow  a  number  of  routes  during 
that  flight.  These  variables  lead  to  a  variety  of  possible  platforms  for  each  asset  and  an 
enonnous  number  of  possible  portfolios. 

Consider  a  universe  where  the  Department  of  Defense  owns  the  following  ISR  assets: 
three  types  of  Remotely  Piloted  Aircraft  (RPA),  two  types  of  piloted  aircraft  (AC),  and  three 
types  of  satellites  (SAT).  These  assets  are  described  in  Table  4.  Two  of  the  RPAs  can  be  flown 
out  of  two  different  bases;  thus  the  number  ’2’  is  placed  in  the  ’Bases'  column  for  RPA1  and 
RPA2  in  Table  4.  The  RPAs  and  ACs  can  also  be  flown  on  different  routes  when  deployed. 
Finally,  several  of  the  ISR  assets  can  feature  multiple  equipment  packages,  as  clarified  by  the 
Platform  Notes.  For  example,  RPA1  and  RPA2  each  can  be  loaded  with  one  of  two  different 
INTI  packages  (INTI a,  INTlb)  or  a  package  that  gathers  INT2  and  INT3  data  (INT2&INT3), 
for  a  total  of  three  different  packages.  AC  1  and  AC2  have  two  possible  package  options:  the 
first  is  designed  for  INT2  but  obtains  some  capability  in  INT3,  while  the  second  is  designed  for 
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INT3  but  obtains  some  capability  in  INT2.  For  initial  simplicity,  this  "some"  capability  can  be 
considered  "half'  capability,  and  will  be  expanded  upon  later.  All  asset/base/route/package 
options  are  summarized  in  Table  4. 


Table  4:  ISR  assets 


Package  Options 

SYSTEM 

Bases 

Routes 

inti 

LNT2 

EST3 

Platform  Notes 

RPA1 

2 

6 

2 

1 

1 

Either  INTI  or  INT2  and  INT3 

RPA2 

2 

6 

2 

1 

1 

Either  INTI  or  INT2  and  ENT 3 

RPA3 

1 

1 

1 

0 

0 

INTI  only 

AC1 

1 

2 

0 

2 

2 

Better  for  EST2  but  can  do  INT3 

AC2 

1 

2 

0 

2 

2 

Better  for  INT3  but  can  do  ENT2 

SAT1 

1 

1 

2 

0 

0 

INTI  only 

SAT2 

1 

1 

0 

1 

0 

INT2  only 

SAT3 

1 

1 

0 

0 

1 

INT3  only 

The  number  of  possible  platform  combinations  for  a  specific  asset  can  then  be  calculated. 
For  example,  RPA1  can  be  stationed  at  two  bases,  perform  six  mission  routes,  and  carry  one  of 
three  packages  (INTI  a,  INTlb,  or  INT2&INT3).  The  number  of  possible  combinations  is  thus 
.  AC  1  can  be  stationed  at  one  base,  perform  two  mission  routes,  and  carry  one  of 
two  packages  (INT2a&INT3a  or  INT2b&INT3b,  with  INT2  achieving  full  capability  and  INT3 
half  capability),  for  a  total  of  four  possible  combinations.  AC2  also  has  four  combinations,  but 
with  INT3  achieving  full  capability  and  INT2  achieving  half  capability.  Following  similar  logic, 
the  number  of  combinations  for  each  asset  is  presented  in  Table  5. 
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Table  5:  ISR  asset  platforms 


Total  Platform 

Combinations 

RPA1 

36 

RPA2 

36 

RPA3 

1 

AC1 

4 

AC2 

4 

SAT1 

2 

SAT2 

1 

SAT3 

1 

Summing  the  individual  options  for  inclusion  in  a  portfolio  results  in  85  possible 
alternatives.  However,  an  asset  can  only  characterize  one  option  at  a  time;  with  only  one  RPA1, 
it  is  not  possible  to  have  two  RPA1  platforms  active  at  once.  To  determine  the  number  of 
portfolios  that  include  all  eight  assets,  the  number  of  platform  combinations  for  each  asset  are 
multiplied  together: 


However,  the  constraints  of  a  selection  problem  may  mean  one  must  consider 
combinations  of  any  number  of  ISR  assets.  Because  the  option  exists  not  to  select  an  ISR  asset, 
an  additional  platform  combination  must  be  added  for  each  asset  —  a  "zero"  option  of  not 
including  an  asset  in  the  portfolio.  Table  6  reflects  this  additional  option  for  each  asset. 
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Table  6:  ISR  asset  options 


Total  Usage 
Options 

RPA1 

37 

RPA2 

37 

RPA3 

2 

AC1 

5 

AC2 

5 

SAT1 

3 

SAT2 

2 

SAT3 

2 

The  number  of  possible  portfolios  of  any  size  (zero  to  eight  assets)  is  much  larger: 

With  only  eight  ISR  assets  and  a  handful  of  variables  for  each,  nearly  a  million  possible 
portfolios  are  generated.  To  evaluate  the  performance  of  each  of  these  portfolios  against  the  63 
measures  in  the  full  hierarchy  requires  significant  processing  time.  Currently,  JFCC-ISR  uses  a 
complex  physics-based  simulation  program  to  evaluate  portfolio  performance.  Given  the  high 
fidelity  of  the  modeling,  processing  time  is  approximately  three  hours  for  a  single  portfolio. 

With  this  rate  of  evaluation,  a  24-processor  computer  network  would  take  — 

hours,  or  1 1.72  years,  to  evaluate  all  possible  portfolios. 

A  slight  modification  to  this  problem  results  in  even  more  extreme  evaluation  difficulty. 
The  current  problem  assumes  one  each  of  the  eight  assets.  However,  instead  of  one  RPA1,  what 
if  there  exist  three  RPA1?  Similarly,  now  there  are  three  RPA2,  and  two  each  of  AC1  and  AC2. 
Certainly,  the  military  possesses  multiple  copies  of  most  ISR  assets,  and  this  example  still 
contains  only  a  few  variables  for  each  asset.  With  only  this  modification,  and  leaving  all  other 
variables  as  they  were,  the  new  usage  options  are  shown  in  Table  7. 
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Table  7:  Multiple  copies  of  an  asset  available 


Total  Usage 
Options 

RPAla 

37 

RPAlb 

37 

RPAlc 

37 

RPA2a 

37 

RPA2b 

37 

RPA2c 

37 

RPA3 

2 

ACla 

5 

AC  lb 

5 

AC2a 

5 

AC2b 

5 

SAT1 

3 

SAT2 

2 

SAT3 

2 

This  modification  results  in  possible  portfolio  combinations.  With  hierarchy 

performance  of  each  portfolio  scored  in  three  hours  by  a  single  processor  running  a  simulation 

program,  a  24-processor  computer  network  would  take  —  hours,  or 

years,  to  evaluate  each  possible  portfolio.  Even  if  the  hierarchy  performance  of  a 
portfolio  could  be  determined  in  one  second,  the  24-processor  computer  network  would  still  take 
50,849  years  to  evaluate  all  possible  portfolios.  Clearly,  even  with  several  factor-of-ten 
advancements  in  computing  power  and  speed,  solving  this  problem  by  evaluating  each  possible 
portfolio  combination  is  impractical.  Evaluating  each  portfolio  for  the  full  military  ISR  problem, 
with  dozens  of  assets  with  multiple  copies  and  hundreds  of  platform  options,  is  impossible. 

Capability  Screening. 

Screening  the  alternatives  to  eliminate  poorly-performing  assets  or  platfonns  which  are 
unlikely  to  add  value  to  a  good  portfolio  can  greatly  reduce  the  total  number  of  portfolios  that 
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must  be  evaluated.  As  previously  explored,  a  small  percentage  decrease  in  the  number  of 
alternatives  in  a  combinatorial  problem  can  lead  to  an  enonnous  percentage  decrease  in  the 
number  of  combinations.  This  can  allow  for  the  transformation  of  a  problem's  full  evaluation 
from  impractical  to  reasonable  and  allow  resources  to  be  better  dedicated  to  refining 
perfonnance  evaluation  of  the  smaller  set. 

Transfonning  the  ISR  performance-based  hierarchy  to  a  capability-based  hierarchy  is  the 
first  step  in  this  screening  process.  This  transformation  will  detennine  which  sub-objectives  and 
associated  measures  of  the  seven  top-level  objectives  are  most  important  through  the  assignment 
of  global  weights.  Then,  the  alternatives  can  be  assessed  on  their  capability  in  each  area,  to 
determine  which  assets  contribute  little  capability  and/or  are  dominated. 

From  the  local  weights  provided  in  the  original  hierarchy,  the  global  weights  for  each 
objective,  sub-objective,  and  measure  can  be  determined,  as  shown  in  Table  8. 
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Table  8:  Objective,  sub-objective,  and  measure  weights 


Local  Weight 

Global  Weight 

Local  Weight 

Global  Weight 

Objective  1 

0.168 

0.168 

Objective  4 

0.074 

0.074 

INT1.1 

1.000 

0.168 

INTI. 4 

0.746 

0.055 

Qual  1.1 

0.397 

0.067 

Qual  1.4 

0.251 

0.014 

Timely  1.1 

0.178 

0.030 

Timely  1.4 

0.350 

0.019 

Persist  1.1 

0.425 

0.071 

Persist  1.4 

0.400 

0.022 

INT2.1 

0.000 

0.000 

INT2.4 

0.055 

0.004 

Qual  2.1 

0.000 

0.000 

Qual  2.4 

0.203 

0.001 

Timely  2.1 

0.000 

0.000 

Timely  2.4 

0.462 

0.002 

Persist  2.1 

0.000 

0.000 

Persist  2.4 

0.335 

0.001 

INT3.1 

0.000 

0.000 

INT3.4 

0.199 

0.015 

Qual  3.1 

0.000 

0.000 

Qual  3.4 

0.203 

0.003 

Timely  3.1 

0.000 

0.000 

Timely  3.4 

0.462 

0.007 

Persist  3.1 

0.000 

0.000 

Persist  3.4 

0.335 

0.005 

Objective  2 

0.039 

0.039 

Objective  5 

0.000 

0.000 

INTI. 2 

1.000 

0.039 

INTI. 5 

0.000 

0.000 

Qual  1.2 

0.251 

0.010 

Qual  1.5 

0.000 

0.000 

Timely  1.2 

0.350 

0.013 

Timely  1.5 

0.000 

0.000 

Persist  1.2 

C.4CC 

0.015 

Persist  1.5 

0.000 

0.000 

INT2.2 

0.000 

0.000 

INT2.5 

0.000 

0.000 

Qual  2.2 

0.000 

0.000 

Qual  2.5 

0.000 

0.000 

Timely  2.2 

0.000 

0.000 

Timely  2.5 

0.000 

0.000 

Persist  2.2 

0.000 

0.000 

Persist  2.5 

0.000 

0.000 

INT3.2 

0.000 

0.000 

INT3.5 

0.000 

0.000 

Qual  3.2 

0.000 

0.000 

Qual  3.5 

0.000 

0.000 

Timely  3.2 

0.000 

0.000 

Timely  3.5 

0.000 

0.000 

Persist  3.2 

0.000 

0.000 

Persist  3.5 

0.000 

0.000 

Objective  3 

0.448 

0.448 

Objective  6 

0.159 

0.159 

INTI. 3 

0.746 

0.334 

INTI. 6 

0.000 

0.000 

Qual  1.3 

0.251 

0.084 

Qual  1.6 

0.000 

0.000 

Timely  1.3 

0.350 

0.117 

Timely  1.6 

0.000 

0.000 

Persist  1.3 

0.400 

0.134 

Persist  1.6 

0.000 

0.000 

INT2.3 

0.055 

0.024 

INT2.6 

0.458 

0.073 

Qual  2.3 

0.203 

0.005 

Qual  2.6 

0.203 

0.015 

Timely  2.3 

0.462 

0.011 

Timely  2.6 

0.462 

0.034 

Persist  2.3 

0.335 

0.008 

Persist  2.6 

0.335 

0.024 

INT3.3 

0.199 

0.089 

INT3.6 

0.542 

0.086 

Qual  3.3 

0.203 

0.018 

Qual  3.6 

0.203 

0.017 

Timely  3.3 

0.462 

0.041 

Timely  3.6 

0.462 

0.040 

Persist  3.3 

0.335 

0.030 

Persist  3.6 

0.335 

0.029 

Objective  7 

0.113 

0.113 

INTI. 7 

0.146 

0.017 

Qual  1.7 

0.251 

0.004 

Timely  1.7 

0.350 

0.006 

Persist  1.7 

0.400 

0.007 

INT2.7 

0.427 

0.048 

Qual  2.7 

0.203 

0.010 

Timely  2.7 

0.462 

0.022 

Persist  2.7 

0.335 

0.016 

INT3.7 

0.427 

0.048 

Qual  3.7 

0.203 

0.010 

Timely  3.7 

0.462 

0.022 

Persist  3.7 

0.335 

0.016 
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Several  of  the  sub-objective  and  measure  global  weights  are  zero;  this  is  an  artifact  of  the 
alternatives  available  when  the  original  JFCC-ISR  hierarchy  was  created,  but  will  not  negatively 
affect  the  validity  of  this  analysis.  By  summing  the  appropriate  global  weights,  the  importance 
of  each  sub-objective  and  measure  can  be  calculated,  as  shown  in  Table  9. 


Table  9:  Sub-objective  and  measure  weights  across  all  objectives 


INTI 

INT2 

INT3 

Quality 

0.178 

0.030 

0.048 

0.257 

Timeliness 

0.185 

0.069 

0.110 

0.365 

Persistence 

0.249 

0.050 

0.080 

0.379 

0.612 

0.150 

0.239 

INTI  is  the  most  important  of  the  three  sub-objectives,  providing  61.2%  of  the  overall 
value  in  all  objectives,  while  INT2  is  least  important  at  15.0%  of  the  value.  The  measures  are 
more  evenly  distributed,  with  Persistence  (37.9%)  and  Timeliness  (36.5%)  most  important  and 
Quality  (25.7%)  the  least. 

A  capability-based  hierarchy  can  be  created  with  these  global  weights,  shown  in  Figure 
14. 


Overall  Goal 
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Or,  simpler  hierarchies  of  only  sub-objectives  (Figure  15)  or  only  measures  (Figure  16)  can  be 
created. 


Figure  15:  Condensed  sub-objective  hierarchy 


Figure  16:  Condensed  measure  hierarchy 


These  hierarchies,  reduced  to  the  sub-objective  and  measure  level,  can  now  be  used  as 
capability-based  hierarchies.  The  sub-objective  capability-based  hierarchy  allows  for  evaluation 
of  the  alternatives  based  on  their  intended  INT  capability.  An  alternative's  score  is  calculated  by 
summing  the  weights  for  each  sub-objective  an  alternative  is  designed  to  achieve. 
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For  example,  RPA1  version  3  carries  a  package  that  gathers  INT2  and  INT3  data,  thus 
providing  capability  in  these  two  sub-objective  areas.  The  weights  of  these  two  areas,  0.150  and 
0.239,  are  summed  to  give  a  capability  score  for  this  alternative  of  0.389.  Table  10  displays 
capability-based  evaluations  of  each  alternative  for  the  initial  case  of  single  copies  of  each  ISR 
asset.  For  each  alternative,  the  grey  shading  represents  in  which  sub-objective  that  alternative  is 
designed  to  provide  capability;  the  number  in  each  cell  specifies  the  base/route/platfonn  option. 
For  AC1  and  AC2,  where  each  platfonn  option  is  designed  to  accomplish  one  sub-objective  well 
and  one  moderately,  a  "half'  factor  of  0.5  was  used  to  represent  this  moderate  capability.  The 
table  is  sorted  from  highest  to  lowest  value  score. 
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Table  10:  Capability  value  scores 


Platform  Variables 


System 

Version 

Base 

Route 

INTI 

INT2 

INT3 

Score 

RPA1 

1 

1 

1 

1 

0 

0 

0.612 

RPA1 

2 

1 

1 

2 

0 

0 

0.612 

RPA1 

4 

1 

2 

1 

0 

0 

0.612 

RPA1 

5 

1 

2 

2 

0 

0 

0.612 

RPA1 

7 

1 

3 

1 

0 

0 

0.612 

RPA1 

8 

1 

3 

2 

0 

0 

0.612 

RPA1 

10 

1 

4 

1 

0 

0 

0.612 

RPA1 

11 

1 

4 

2 

0 

0 

0.612 

RPA1 

13 

1 

5 

1 

0 

0 

0.612 

RPA1 

14 

1 

5 

2 

0 

0 

0.612 

RPA1 

16 

1 

6 

1 

0 

0 

0.612 

RPA1 

17 

1 

6 

2 

0 

0 

0.612 

RPA1 

19 

2 

1 

1 

0 

0 

0.612 

RPA1 

20 

2 

1 

2 

0 

0 

0.612 

RPA1 

22 

2 

2 

1 

0 

0 

0.612 

RPA1 

23 

2 

2 

2 

0 

0 

0.612 

RPA1 

25 

2 

3 

1 

0 

0 

0.612 

RPA1 

26 

2 

3 

2 

0 

0 

0.612 

RPA1 

28 

2 

4 

1 

0 

0 

0.612 

RPA1 

29 

2 

4 

2 

0 

0 

0.612 

RPA1 

31 

2 

5 

1 

0 

0 

0.612 

RPA1 

32 

2 

5 

2 

0 

0 

0.612 

RPA1 

34 

2 

6 

1 

0 

0 

0.612 

RPA1 

35 

2 

6 

2 

0 

0 

0.612 

RPA2 

1 

1 

1 

1 

0 

0 

0.612 

RPA2 

2 

1 

1 

2 

0 

0 

0.612 

RPA2 

4 

1 

2 

1 

0 

0 

0.612 

RPA2 

5 

1 

2 

2 

0 

0 

0.612 

RPA2 

7 

1 

3 

1 

0 

0 

0.612 

RPA2 

8 

1 

3 

2 

0 

0 

0.612 

RPA2 

10 

1 

4 

1 

0 

0 

0.612 

RPA2 

n _ 

1 

4 

2 

0 

0 

0.612 

RPA2 

13 

1 

5 

1 

0 

0 

0.612 

RPA2 

14 

1 

5 

2 

0 

0 

0.612 

RPA2 

16 

1 

6 

1 

0 

0 

0.612 

RPA2 

17 

1 

6 

2 

0 

0 

0.612 

RPA2 

19 

2 

1 

1 

0 

0 

0.612 

RPA2 

20 

2 

1 

2 

0 

0 

0.612 

RPA2 

22 

2 

2 

1 

0 

0 

0.612 

RPA2 

23 

2 

2 

2 

0 

0 

0.612 

RPA2 

25 

2 

3 

1 

0 

0 

0.612 

RPA2 

26 

2 

3 

2 

0 

0 

0.612 

RPA2 

28 

2 

4 

1 

0 

0 

0.612 

RPA2 

29 

2 

4 

2 

0 

0 

0.612 

RPA2 

31 

2 

5 

1 

0 

0 

0.612 

RPA2 

32 

2 

5 

2 

0 

0 

0.612 

Platform  Variables 

System 

Version 

Base 

Route 

INTI 

INT2 

INT3 

Score 

RPA2 

34 

2 

6 

1 

0 

0 

0.612 

RPA2 

35 

2 

6 

2 

0 

0 

0.612 

RPA3 

1 

1 

1 

1 

0 

0 

0.612 

SAT1 

1 

1 

1 

1 

0 

0 

0.612 

SAT1 

2 

1 

1 

2 

0 

0 

0.612 

RPA1 

3 

1 

1 

0 

1 

1 

0.389 

RPA1 

6 

1 

2 

0 

1 

1 

0.389 

RPA1 

9 

1  | 

3 

0 

1 

1 

0.389 

RPA1 

12 

1 

4 

0 

1 

1 

0.389 

RPA1 

15 

1 

5 

0 

1 

1 

0.389 

RPA1 

18 

1 

6 

0 

1 

1 

0.389 

RPA1 

21 

2 

1 

0 

1 

1 

0.389 

RPA1 

24 

2 

2 

0 

1 

1 

0.389 

RPA1 

27 

2 

3 

0 

1 

1 

0.389 

RPA1 

30 

2 

4 

0 

1 

1 

0.389 

RPA1 

33 

2 

5 

0 

1 

1 

0.389 

RPA1 

36 

2 

6 

0 

1 

1 

0.389 

RPA2 

3 

1 

1 

0 

1 

1 

0.389 

RPA2 

6 

1 

2 

0 

1 

1 

0.389 

RPA2 

9 

1 

3 

0 

1 

1 

0.389 

RPA2 

12 

1 

4 

0 

1 

1 

0.389 

RPA2 

15 

~  1  | 

5 

0 

1 

1 

0.389 

RPA2 

18 

1 

6 

0 

1 

1 

0.389 

RPA2 

21 

2 

1 

0 

1 

1 

0.389 

RPA2 

24 

2 

2 

0 

1 

1 

0.389 

RPA2 

27 

2 

3 

0 

1 

1 

0.389 

RPA2 

30 

~~  2  j 

4 

0 

1 

1 

0.389 

RPA2 

33 

2 

5 

0 

1 

1 

0.389 

RPA2 

36 

2 

6 

0 

1 

1 

0.389 

AC2 

1 

1 

1 

0 

0.5 

1 

0.314 

AC2 

2 

1  | 

1 

0 

0.5 

1 

0.314 

AC2 

3 

1 

2 

0 

0.5 

1 

0.314 

AC2 

4 

1 

2 

0 

0.5 

1 

0.314 

AC1 

1 

1 

1 

0 

1 

0.5 

0.270 

AC1 

2 

1 

1 

0 

1 

0.5 

0.270 

AC1 

3 

1 

2 

0 

1 

0.5 

0.270 

AC1 

4 

1 

2 

0 

1 

0.5 

0.270 

SAT3 

1 _ 

1 

1 

0 

0 

1 

0.239 

SAT2 

1 

1 

1 

0 

1 

0 

0.150 

RPA1 

0 

0 

0 

0 

0 

0 

0.000 

RPA2 

0 

0 

0 

0 

oj 

0 

0.000 

RPA3 

0 

0 

0 

0 

0 

0 

0.000 

AC1 

0 

0 

0 

0 

0 

0 

0.000 

AC2 

0 

0 

0 

0 

0 

0 

0.000 

SAT1 

0 

0 

0 

0 

0 

0 

0.000 

SAT2 

0 

r~  o  | 

0 

0 

0 

0 

0.000 

SAT3 

0 

0 

0 

0 

0 

0 

0.000 

The  assets  that  provide  INT 1  capability  are  most  valued,  followed  by  those  that  provide 
INT2  and  INT3  in  various  combinations.  Finally,  the  assets  that  provide  only  INT2  or  only 
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INT3  round  out  the  bottom  of  the  alternatives  that  provide  value.  Of  course,  the  "zero  option"  of 
not  using  an  ISR  asset  results  in  zero  value  added. 

Now  that  alternative  capability  scores  have  been  calculated,  initial  screening  can  begin. 
The  rough  screening  process  described  next  creates  a  structure  that  will  be  built  on  further. 
Depending  on  the  constraints  of  the  portfolio  selection  problem,  some  alternatives  may  be 
eliminated  based  on  pure  dominance.  This  elimination  of  alternatives  based  solely  on  capability 
scores  is  an  aggressive  screening  method,  and  should  be  used  with  caution.  Problems  with 
alternatives  of  clearly  binary  capability,  without  significant  shades  of  perfonnance,  may  be 
appropriate  for  this  method.  More  refined  screening  methods  will  be  described  later. 

If  one  is  allowed  to  select  any  number  of  assets,  naturally  one  would  choose  all  eight 
assets  for  maximum  capability.  However,  if  a  constraint  exists  that  states  only  a  subset  of  the 
total  number  of  assets  may  be  selected,  then  an  asset  that  is  dominated  by  another  asset  in  a 
certain  set  of  sub -objectives  can  be  removed.  Suppose  one  wishes  to  maximize  total  value  but 
also  to  achieve  value  in  all  three  INT  areas,  and  only  two  of  the  eight  assets  can  be  selected.  An 
asset  that  provides  less  capability  than  another  asset  in  a  specific  sub-objective  set  can  be 
eliminated  through  total  dominance.  In  this  case,  all  versions  of  AC2  (0.314)  and  AC1  (0.270), 
as  well  as  SAT3  (0.239)  and  SAT2  (0.150),  are  dominated  in  INT2  and  INT3  by  the  "INT2  & 
INT3"  versions  of  RPA1  (0.389)  and  RPA2  (0.389).  Thus,  one  would  pick  an  INTI  package 
among  RPA1,  RPA2,  RPA3,  and  SAT1,  and  an  "INT2  &  INT3"  package  from  RPA1  or  RPA2. 
Note  that  RPA1  or  RPA2  can  only  be  selected  to  fulfill  one  of  those  sub-objective  sets,  as  in  this 
example  there  is  only  one  copy  available  of  each  asset.  If  one  chooses  RPA  1  for  INTI,  one 
could  not  choose  it  also  for  "INT2  &  INT3." 
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This  eliminates  10  alternatives  from  the  set,  reducing  the  number  of  alternatives  for  a 


potential  portfolio  from  93  to  83,  as  shown  in  Table  11. 


Table  11:  Alternative  reduction 


System 

Total  Platforms 

After  Screening 

RPA1 

37 

37 

RPA2 

37 

37 

RPA3 

2 

2 

AC1 

5 

1 

AC2 

5 

1 

SAT1 

3 

3 

SAT2 

2 

1 

SAT3 

2 

1 

However,  the  time  to  solve  is  greatly  improved,  as  seen  in  Table  12. 


Table  12:  Combination  reduction 


Original 

After  Screening 

%  Reduction 

Total  Alternatives 

93 

83 

10.75% 

Total  Combinations 

821400 

8214 

99.00% 

Time  to  solve  (days) 

4278.1 

42.8 

99.00% 

Reducing  the  number  of  alternatives  by  10.75%  through  removing  dominated  assets 
reduces  the  total  number  of  portfolios,  and  thus  the  time  to  solve,  by  99%.  Table  12  shows  time 
to  solve  with  a  24-processor  network  when  each  portfolio  is  processed  in  three  hours.  This 
processing  time  definition  is  arbitrary;  the  percent  reduction  in  total  combinations  (and  thus 
reduction  in  time  to  solve,  no  matter  the  unit)  is  99%,  an  incredible  savings.  As  the  type  of 
constraint  changes,  such  as  number  of  allowable  assets,  more  or  fewer  assets  can  be  eliminated, 
adjusting  the  percent  reduction  in  time  to  solve. 
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This  basic  screening  can  be  used  to  account  for  other  constraints.  If  one  type  of  aircraft 
must  be  chosen,  it  would  be  advantageous  to  choose  AC2  (0.3 14)  over  AC  1  (0.270),  as  AC2  is 
stronger  in  INT3,  which  is  weighted  heavier  (and  thus  contributes  more  value  to  the  full 
hierarchy)  than  ACl's  INT2  strength.  If  two  of  the  three  satellites  must  be  chosen,  the  best 
combination  is  SAT1  (0.612)  and  SAT3  (0.239).  These  types  of  alternative-selection  problems 
can  be  solved  using  standard  mixed  integer  programming  techniques  with  simple  constraints. 

Other  times,  inherent  properties  are  known  about  two  similar  ISR  assets.  For  example, 
consider  RPA1  and  RPA2.  Each  has  identical  basing,  route,  and  platform  options.  It  may  be 
known  that  RPA1  is  a  newer,  updated  version  of  RPA2.  Thus,  it  can  be  assumed  that  RPA1, 
though  it  is  designed  to  have  the  same  capability  set  as  RPA2,  actually  performs  at  a  better  level 
operationally.  Exact  perfonnance  data  is  not  needed;  the  knowledge  that  as  RPA1  was  created  to 
improve  upon  RPA2  allows  for  the  assumption  that  RPA1  performs  better  than  RPA2. 

Whenever  an  RPA1  platfonn  provides  the  identical  sub-objective  capability  set  as  RPA2,  the 
RPA2  platform  can  be  eliminated  as  its  performance  is  assumed  to  be  dominated.  (In  a  more 
rigorous  analysis  it  might  be  necessary  to  consider  other  factors  such  as  cost  of  operation,  which 
may  dissuade  elimination.)  In  the  situation  under  examination,  RPA1  and  RPA2  both  provide 
three  possible  platfonn  configurations  (two  versions  of  INT 1  plus  one  version  of  "INT2  & 
INT3").  If  a  constraint  exists  that  only  one  RPA  asset  may  be  selected,  because  it  is  assumed 
RPA1  provides  better  inherent  capability  than  RPA2,  RPA2  can  be  eliminated  from  the 
alternative  set.  This  results  in  the  post-screening  alternative  list  in  Table  13. 
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Table  13:  Alternative  reduction 


System 

Total  Platforms 

After  Screening 

RPA1 

37 

37 

RPA2 

37 

1 

RPA3 

2 

2 

AC1 

5 

5 

AC2 

5 

5 

SAT1 

3 

3 

SAT2 

2 

2 

SAT3 

2 

2 

This  type  of  logical  screening,  possible  during  capability-based  examination,  can  greatly 
reduce  the  number  of  potential  portfolio  combinations  as  shown  in  Table  14. 


Table  14:  Combination  reduction 


Original 

After  Screening 

%  Reduction 

Total  Alternatives 

93 

57 

38.71% 

Total  Combinations 

821400 

22200 

97.30% 

Time  to  solve  (days) 

4278.1 

115.6 

97.30% 

Eliminating  RPA2  from  consideration  reduces  the  total  number  of  alternatives  by 
38.71%,  and  reduces  the  number  of  portfolios  (and  thus  time  to  solve)  by  97.30%. 

Note  that  in  this  case,  38.71%  of  alternatives  eliminated  produced  a  lower  percentage 
reduction  in  total  combinations  (97.30%)  than  in  the  previous  case  when  10.75%  of  alternatives 
were  eliminated  (99.00%).  When  eliminating  alternatives  from  one  asset,  the  percent  reduction 
in  alternatives  can  be  much  higher  (38.71%  vs.  10.75%),  but  the  percent  reduction  in  total 
combinations  may  be  lower  (97.30%  vs.  99.00%).  This  is  a  function  of  the  multiplicative  nature 
of  calculating  total  combinations.  The  first  case  removed  a  multiplicative  factor  of  100  through 
the  removal  of  10  alternatives:  AC1  platforms  from  five  to  one  (removal  of  multiplicative  factor 
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of  5),  AC2  from  five  to  one  (5),  SAT2  from  two  to  one  (2),  and  SAT3  from  two  to  one  (2),  and 

The  second  case  removed  a  multiplicative  factor  of  37  through  the 
elimination  of  36  alternatives.  Thus,  it  is  generally  better  to  screen  and  eliminate  platforms  from 
multiple  assets  than  only  to  remove  a  large  number  of  platforms  from  one  asset.  The  number  of 
alternatives  eliminated  is  not  a  perfect  indicator  of  percentage  decrease  in  the  number  of 
combinations. 

Advanced  Screening. 

The  binary  capability-based  screening  process  previously  described  is  useful  for 
eliminating  totally  dominated  alternatives.  However,  many  times  an  alternative  will  not  be 
completely  dominated  by  another  alternative;  organizational  efficiencies  tend  to  eliminate 
redundant  assets.  It  may  be  useful  to  perfonn  a  more  detailed  procedure  that  is  fundamentally 
capability-based  but  adds  a  slight  layer  of  estimated  alternative  performance  for  each  capability. 
This  process  requires  more  resources  initially,  but  can  be  accomplished  in  an  entirely  reasonable 
amount  of  time  and  greatly  reduces  the  time  and  cost  necessary  for  later  detailed  portfolio 
performance-based  analysis. 

This  new  process  uses  the  rolled-up  hierarchy  that  consolidated  sub-objectives  and 
measures  for  each  objective,  as  shown  in  Figure  14.  This  level  of  granularity  may  be  necessary 
if  the  measures  of  Quality,  Timeliness,  and  Persistence  cannot  be  generalized  across  all  sub¬ 
objectives.  An  alternative  cannot  be  given  a  single  score  to  represent  "quality,"  because  quality 
in  the  INTI  domain  may  be  quite  different  than  quality  in  the  INT3  domain.  Similarly,  beyond 
the  most  basic  capability-based  analysis,  it  is  not  good  practice  to  give  an  alternative  a  binary 
score  for  capability  in  a  sub-objective.  For  example,  an  asset  may  be  designed  to  provide 
capability  in  INT3,  but  achieves  this  through  extreme  strength  in  Quality  3.  Because  Quality  3  is 
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a  low-weighted  measure  for  INT3,  that  asset  is  really  only  achieving  l/6th  of  the  possible  INT3 
value. 

A  better  primary  analysis  can  yield  much  better  infonnation  on  alternative  value,  and  thus 
which  alternatives  may  comprise  better  portfolios.  Establishing  quality  alternatives  through 
primary  analysis  allows  for  the  confident  elimination  of  poor  alternatives,  thus  reducing  the 
number  of  portfolios.  Then,  better  starting  points  for  a  response  surface  methodology 
algorithmic  search  in  conjunction  with  the  detailed  physics-based  computer  portfolio  simulations 
can  be  created. 

To  begin,  each  of  the  nine  measures  is  assigned  a  value  function  that  maps  an 
alternative's  estimated  capability  level  in  a  measure  to  the  value  of  that  capability  level.  These 
value  functions  are  operational  for  every  alternative,  so  they  are  allowed  to  be  broad,  even 
categorical  if  necessary,  to  account  for  the  wide  range  of  capabilities  and  levels  of  performance 
ISR  assets  provide.  Because  these  value  functions  operate  on  the  condensed  hierarchy,  they  are 
defined  by  the  capability  range  of  the  alternative  set  itself,  not  for  any  specific  objective  domain. 

Once  the  nine  value  functions  are  created,  the  subject  matter  expert  for  each  asset  scores 
each  of  his  asset's  platform  alternatives  in  each  measure.  Each  asset  alternative  may  achieve 
different  value  in  a  measure  based  on  its  platfonn  options  (i.e.,  base,  route,  ISR  package).  The 
asset  subject  matter  expert  estimates  each  platform's  capability  level,  which  is  then  mapped  to  a 
value  through  the  value  functions.  This  task  should  not  present  significant  difficulty  to  a  party 
familiar  with  an  asset's  capabilities  under  different  conditions.  Each  asset  is  scored  by  a  different 
subject  matter  expert,  so  the  total  work  is  spread  among  a  large  number  of  people.  This  may 
prove  the  lengthiest  part  of  the  process  in  terms  of  man-hours,  but  the  work  can  be  done 
concurrently  and  adds  tremendous  value  to  the  fidelity  of  this  screening  methodology. 


69 


Expanding  on  the  ISR  example  explored  previously,  let  the  measure  value  functions  be 


defined  categorically  as  shown  in  Table  15. 


Table  15:  Categorical  value  functions 


Value 

INTI 

Definiton 

Low 

Medium 

High 

Quality  1 

resolution 

0.10 

0.60 

0.95 

Timeliness  1 

transmission  time 

0.30 

0.60 

0.90 

Persistence  1 

collections  per  day 

0.40 

0.60 

0.80 

INT2 

Quality  2 

minimum  signal/noise  ratio 

0.40 

0.50 

0.75 

Timeliness  2 

transmission  time 

0.60 

0.70 

0.80 

Persistence  2 

time  in  operation 

0.55 

0.75 

0.95 

INT3 

Quality  3 

minimum  signal/noise  ratio 

0.20 

0.60 

0.90 

Timeliness  3 

transmission  time 

0.20 

0.70 

0.95 

Persistence  3 

time  in  operation 

0.30 

0.80 

0.95 

The  definitions  for  INT2  and  INT3  are  identical  because  these  methods  of  intelligence 
collection  are  similar  in  nature.  The  value  functions  differ  because  the  distribution  of  alternative 
capability  level  in  a  measure  is  different  between  the  two  sub-objectives,  as  is  the  value  to  a  sub¬ 
objective  of  certain  levels  of  capability. 

An  ISR  alternative  that  provides  low  resolution  for  INTI  achieves  a  value  of  0.10  for  the 
INTI  Quality  measure.  (Actual  low,  medium,  and  high  capability  level  definitions  for  each 
measure  are  available  but  classified  and  not  included  here.)  Multiplying  this  value  by  the  global 
weight  of  the  INTI  Quality  measure  (0.178)  gives  0.0178  as  the  weighted  measure  value 
achieved.  This  process  is  repeated  for  each  of  the  nine  measures,  and  the  sum  of  the  weighted 
measure  values  provides  the  score  for  that  alternative. 
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The  subject  matter  experts  evaluate  the  set  of  nine  measures  for  each  of  an  asset's 
platforms.  While  it  seems  a  large  number  of  evaluations  must  occur,  with  categorical  value 
functions,  many  platfonns  are  likely  to  have  identical  value  in  certain  measures.  Additionally, 
many  of  the  measures  will  simply  be  assigned  a  value  of  zero  (not  low)  if  a  platfonn  provides  no 
capability  in  that  sub-objective,  thus  simplifying  the  process.  It  is  assumed  that  no  platfonn  is 
perfect  and  thus  there  is  no  option  for  a  value  score  of  one.  If  an  "ideal"  platform  can  be  defined, 
it  can  be  included  as  a  category  with  value  one. 

The  assets  under  consideration  in  this  example  vary  in  sub-objective  performance  as  a 
function  of  base,  route,  and  package.  Persistence  signifies  the  quantity  of  data  the  ISR  asset 
provides,  which  is  commonly  a  function  of  the  amount  of  time  spent  collecting  data.  Persistence 
can  be  affected  by  the  route  a  asset  takes,  as  some  flight  paths  may  provide  a  longer  duration  of 
target  coverage.  The  RPAs  score  high  in  persistence  on  routes  1  and  2,  which  are  direct  paths  to 
the  target  area.  The  RPAs  score  low  in  persistence  for  routes  5  and  6,  which  require  a  longer 
travel  path  until  the  RPA  nears  the  target;  only  a  short  loiter  time  far  from  the  target  is  possible 
before  the  RPA  must  return  for  fuel.  RPA  routes  3  and  4  fall  between  these  extremes.  The  ACs 
can  follow  two  possible  routes:  route  1  allows  for  direct  travel  and  high  persistence,  while  route 
2  requires  a  longer  path  and  thus  only  medium  persistence  before  the  aircraft  must  return  to  base. 
The  three  SATs  are  always  in  geosynchronous  orbit  and  are  capable  of  constant  surveillance; 
thus,  high  persistence. 

Timeliness  refers  to  the  speed  at  which  an  ISR  asset  can  provide  raw  data  to  the  analysis 
center  for  transfonnation  into  useable  information,  and  thus  is  a  function  of  the  asset.  RPA1 
possesses  high  timeliness  due  to  upgraded  software  and  hardware  assets  that  allow  for  powerful 
compression  and  fast  transmission.  RPA2  and  RPA3,  as  well  as  AC1  and  AC2,  utilize  older 
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technology  that  cannot  transmit  as  efficiently,  and  thus  exhibit  medium  timeliness.  Naturally, 
the  ever-present  SATs  also  possess  high  timeliness. 

Quality  is  a  function  of  asset,  route,  and  the  specific  ISR  package  installed  on  the  asset. 
The  newer  RPA1  has  better  technology,  while  RPA2  and  RPA3  have  older  packages  installed. 
Routes  2  and  4  bring  an  RPA  closer  to  the  target  area,  leading  to  higher  quality  data,  but  there  is 
a  risk  of  interference  from  the  enemy.  Routes  1  and  3  keep  an  RPA  safer  by  following  a  path 
farther  from  the  target  area,  but  thus  lead  to  lower  quality  data.  So,  RPA1  following  route  2  or  4 
achieves  high  quality,  while  route  1  or  3  leads  to  medium  quality.  For  RPA2,  route  2  or  4 
achieves  medium  quality,  while  route  1  or  3  leads  to  low  quality.  Routes  5  and  6  are  far  from  the 
target  area  and  thus  have  low  quality  for  all  RPAs.  AC1  can  carry  a  high  quality  INT2  package 
and  medium  quality  INT3  package,  while  AC2  can  carry  medium  quality  INT2  and  high  quality 
INT3;  their  packages  are  such  that  route  does  not  change  performance  enough  to  move  value 
function  category.  The  SATs  all  offer  high  quality  capabilities. 

With  the  values  achieved  in  each  measure  established,  an  alternative  can  be  scored  by 
multiplying  the  value  for  a  measure  by  the  global  weight  of  that  measure.  For  example,  scoring 
for  RPA1  version  3  is  shown  in  Table  16. 


Table  16:  Scoring  for  an  alternative 


INTI 

INT2 

INT3 

Option 

Quality  1  Timeliness  1  Persistence  1 

Quality  2 

Timeliness  2  Persistence  2 

Quality  3 

Timeliness  3 

Persistence  3 

System  Version 

Base  Route  INTI  INT2 

INT3  0.178  0.185  0.249 

0.030 

0.069  0.050 

0.048 

0.110 

0.080 

Total  Score 

RPA1  3 

110  1 

10  0  0 

0.50 

0.80  0.95 

0.60 

0.95 

0.95 

0.3270 

This  platform  achieves  value  of  0.50  for  Quality  2,  0.80  for  Timeliness  2,  and  0.95  for 


Persistence  2,  plus  0.60  for  Quality  3,  0.95  for  Timeliness  3,  and  0.95  for  Persistence  3. 
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Multiplying  each  value  by  its  respective  weight  (found  in  Table  16  row  3  under  the  measure 
name)  and  summing  results  in  a  total  score  of  0.3270. 

Table  17  shows  the  list  of  scored  alternatives  sorted  by  total  value  achieved. 


Table  17:  Scoring  for  all  alternatives 


INTI 

INT2 

INT3 

Option 

Quality  1  Timeliness  1 

Persistence  1 

Quality  2  Timeliness  2 

Persistence  2 

Quality  3 

Timeliness  3 

Persistence  3 

System  Version 

Base 

Route 

INTI 

INT2 

INT3 

0.178 

0.185 

0.249 

0.030 

0.069 

0.050 

0.048 

0.110 

0.080 

Total  Score 

RPA1  4 

1 

2 

1 

0 

0 

0.95 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.5348 

RPA1  5 

1 

2 

2 

0 

0 

0.95 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.5348 

RPA1  22 

2 

2 

1 

0 

0 

0.95 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.5348 

RPA1  23 

2 

2 

2 

0 

0 

0.95 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.5348 

SAT1  1 

1 

1 

1 

0 

0 

0.95 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.5348 

SAT1  2 

1 

1 

2 

0 

0 

0.95 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.5348 

RPA1  10 

1 

4 

1 

0 

0 

0.95 

0.90 

0.60 

0 

0 

0 

0 

0 

0 

0.4850 

RPA1  11 

1 

4 

2 

0 

0 

0.95 

0.90 

0.60 

0 

0 

0 

0 

0 

0 

0.4850 

RPA1  28 

2 

4 

1 

0 

0 

0.95 

0.90 

0.60 

0 

0 

0 

0 

0 

0 

0.4850 

RPA1  29 

2 

4 

2 

0 

0 

0.95 

0.90 

0.60 

0 

0 

0 

0 

0 

0 

0.4850 

RPA2  4 

1 

2 

1 

0 

0 

0.95 

0.60 

0.80 

0 

0 

0 

0 

0 

0 

0.4793 

RPA2  5 

1 

2 

2 

0 

0 

0.95 

0.60 

0.80 

0 

0 

0 

0 

0 

0 

0.4793 

RPA2  22 

2 

2 

1 

0 

0 

0.95 

0.60 

0.80 

0 

0 

0 

0 

0 

0 

0.4793 

RPA2  23 

2 

2 

2 

0 

0 

0.95 

0.60 

0.80 

0 

0 

0 

0 

0 

0 

0.4793 

RPA1  1 

1 

1 

1 

0 

0 

0.60 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.4725 

RPA1  2 

1 

1 

2 

0 

0 

0.60 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.4725 

RPA1  19 

2 

1 

1 

0 

0 

0.60 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.4725 

RPA1  20 

2 

1 

2 

0 

0 

0.60 

0.90 

0.80 

0 

0 

0 

0 

0 

0 

0.4725 

RPA2  10 

1 

4 

1 

0 

0 

0.95 

0.60 

0.60 

0 

0 

0 

0 

0 

0 

0.4295 

RPA2  11 

1 

4 

2 

0 

0 

0.95 

0.60 

0.60 

0 

0 

0 

0 

0 

0 

0.4295 

RPA2  28 

2 

4 

1 

0 

0 

0.95 

0.60 

0.60 

0 

0 

0 

0 

0 

0 

0.4295 

RPA2  29 

2 

4 

2 

0 

0 

0.95 

0.60 

0.60 

0 

0 

0 

0 

0 

0 

0.4295 

RPA1  7 

1 

3 

1 

0 

0 

0.60 

0.90 

0.60 

0 

0 

0 

0 

0 

0 

0.4227 

RPA1  8 

1 

3 

2 

0 

0 

0.60 

0.90 

0.60 

0 

0 

0 

0 

0 

0 

0.4227 

RPA1  25 

2 

3 

1 

0 

0 

0.60 

0.90 

0.60 

0 

0 

0 

0 

0 

0 

0.4227 

RPA1  26 

2 

3 

2 

0 

0 

0.60 

0.90 

0.60 

0 

0 

0 

0 

0 

0 

0.4227 

RPA2  1 

1 

1 

1 

0 

0 

0.60 

0.60 

0.80 

0 

0 

0 

0 

0 

0 

0.4170 

RPA2  2 

1 

1 

2 

0 

0 

0.60 

0.60 

0.80 

0 

0 

0 

0 

0 

0 

0.4170 

RPA2  19 

2 

1 

1 

0 

0 

0.60 

0.60 

0.80 

0 

0 

0 

0 

0 

0 

0.4170 

RPA2  20 

2 

1 

2 

0 

0 

0.60 

0.60 

0.80 

0 

0 

0 

0 

0 

0 

0.4170 

RPA3  1 

1 

1 

1 

0 

0 

0.60 

0.60 

0.80 

0 

0 

0 

0 

0 

0 

0.4170 

RPA2  7 

1 

3 

1 

0 

0 

0.60 

0.60 

0.60 

0 

0 

0 

0 

0 

0 

0.3672 

RPA2  8 

1 

3 

2 

0 

0 

0.60 

0.60 

0.60 

0 

0 

0 

0 

0 

0 

0.3672 

RPA2  25 

2 

3 

1 

0 

0 

0.60 

0.60 

0.60 

0 

0 

0 

0 

0 

0 

0.3672 

RPA2  26 

2 

3 

2 

0 

0 

0.60 

0.60 

0.60 

0 

0 

0 

0 

0 

0 

0.3672 

RPA1  6 

1 

2 

0 

1 

1 

0 

0 

0 

0.75 

0.80 

0.95 

0.90 

0.95 

0.95 

0.3489 

RPA1  24 

2 

2 

0 

1 

1 

0 

0 

0 

0.75 

0.80 

0.95 

0.90 

0.95 

0.95 

0.3489 

RPA1  3 

1 

1 

0 

1 

1 

0 

0 

0 

0.50 

0.80 

0.95 

0.60 

0.95 

0.95 

0.3270 

RPA1  21 

2 

1 

0 

1 

1 

0 

0 

0 

0.50 

0.80 

0.95 

0.60 

0.95 

0.95 

0.3270 

RPA1  12 

1 

4 

0 

1 

1 

0 

0 

0 

0.75 

0.80 

0.75 

0.90 

0.95 

0.80 

0.3269 

RPA1  30 

2 

4 

0 

1 

1 

0 

0 

0 

0.75 

0.80 

0.75 

0.90 

0.95 

0.80 

0.3269 

RPA2  6 

1 

2 

0 

1 

1 

0 

0 

0 

0.75 

0.70 

0.95 

0.90 

0.70 

0.95 

0.3145 

RPA2  24 

2 

2 

0 

1 

1 

0 

0 

0 

0.75 

0.70 

0.95 

0.90 

0.70 

0.95 

0.3145 

RPA1  9 

1 

3 

0 

1 

1 

0 

0 

0 

0.50 

0.80 

0.75 

0.60 

0.95 

0.80 

0.3050 

RPA1  27 

2 

3 

0 

1 

1 

0 

0 

0 

0.50 

0.80 

0.75 

0.60 

0.95 

0.80 

0.3050 

AC2  1 

1 

1 

0 

1 

1 

0 

0 

0 

0.50 

0.70 

0.95 

0.90 

0.70 

0.80 

0.2950 

AC2  2 

1 

1 

0 

2 

2 

0 

0 

0 

0.50 

0.70 

0.95 

0.90 

0.70 

0.80 

0.2950 

RPA2  3 

1 

1 

0 

1 

1 

0 

0 

0 

0.50 

0.70 

0.95 

0.60 

0.70 

0.95 

0.2926 

RPA2  21 

2 

1 

0 

1 

1 

0 

0 

0 

0.50 

0.70 

0.95 

0.60 

0.70 

0.95 

0.2926 

RPA2  12 

1 

4 

0 

1 

1 

0 

0 

0 

0.75 

0.70 

0.75 

0.90 

0.70 

0.80 

0.2925 

RPA2  30 

2 

4 

0 

1 

1 

0 

0 

0 

0.75 

0.70 

0.75 

0.90 

0.70 

0.80 

0.2925 

AC1  1 

1 

1 

0 

1 

1 

0 

0 

0 

0.75 

0.70 

0.95 

0.60 

0.70 

0.80 

0.2881 

AC1  2 

1 

1 

0 

2 

2 

0 

0 

0 

0.75 

0.70 

0.95 

0.60 

0.70 

0.80 

0.2881 

AC2  3 

1 

2 

0 

1 

1 

0 

0 

0 

0.50 

0.70 

0.75 

0.90 

0.70 

0.80 

0.2850 

AC2  4 

1 

2 

0 

2 

2 

0 

0 

0 

0.50 

0.70 

0.75 

0.90 

0.70 

0.80 

0.2850 

RPA1  13 

1 

5 

1 

0 

0 

0.10 

0.90 

0.40 

0 

0 

0 

0 

0 

0 

0.2839 

RPA1  14 

1 

5 

2 

0 

0 

0.10 

0.90 

0.40 

0 

0 

0 

0 

0 

0 

0.2839 

RPA1  16 

1 

6 

1 

0 

0 

0.10 

0.90 

0.40 

0 

0 

0 

0 

0 

0 

0.2839 

RPA1  17 

1 

6 

2 

0 

0 

0.10 

0.90 

0.40 

0 

0 

0 

0 

0 

0 

0.2839 

RPA1  31 

2 

5 

1 

0 

0 

0.10 

0.90 

0.40 

0 

0 

0 

0 

0 

0 

0.2839 

RPA1  32 

2 

5 

2 

0 

0 

0.10 

0.90 

0.40 

0 

0 

0 

0 

0 

0 

0.2839 

RPA1  34 

2 

6 

1 

0 

0 

0.10 

0.90 

0.40 

0 

0 

0 

0 

0 

0 

0.2839 

RPA1  35 

2 

6 

2 

0 

0 

0.10 

0.90 

0.40 

0 

0 

0 

0 

0 

0 

0.2839 

73 


AC1  3 

1 

2 

0 

1 

1 

0 

0 

0 

0.75 

0.70 

0.75 

0.60 

0.70 

0.80 

0.2781 

AC1  4 

1 

2 

0 

2 

2 

0 

0 

0 

0.75 

0.70 

0.75 

0.60 

0.70 

0.80 

0.2781 

RPA2  9 

1 

3 

0 

1 

1 

0 

0 

0 

0.50 

0.70 

0.75 

0.60 

0.70 

0.80 

0.2706 

RPA2  27 

2 

3 

0 

1 

1 

0 

0 

0 

0.50 

0.70 

0.75 

0.60 

0.70 

0.80 

0.2706 

RPA1  15 

1 

5 

0 

1 

1 

0 

0 

0 

0.40 

0.80 

0.55 

0.20 

0.95 

0.30 

0.2328 

RPA1  18 

1 

6 

0 

1 

1 

0 

0 

0 

0.40 

0.80 

0.55 

0.20 

0.95 

0.30 

0.2328 

RPA1  33 

2 

5 

0 

1 

1 

0 

0 

0 

0.40 

0.80 

0.55 

0.20 

0.95 

0.30 

0.2328 

RPA1  36 

2 

6 

0 

1 

1 

0 

0 

0 

0.40 

0.80 

0.55 

0.20 

0.95 

0.30 

0.2328 

RPA2  13 

1 

5 

1 

0 

0 

0.10 

0.60 

0.40 

0 

0 

0 

0 

0 

0 

0.2284 

RPA2  14 

1 

5 

2 

0 

0 

0.10 

0.60 

0.40 

0 

0 

0 

0 

0 

0 

0.2284 

RPA2  16 

1 

6 

1 

0 

0 

0.10 

0.60 

0.40 

0 

0 

0 

0 

0 

0 

0.2284 

RPA2  17 

1 

6 

2 

0 

0 

0.10 

0.60 

0.40 

0 

0 

0 

0 

0 

0 

0.2284 

RPA2  31 

2 

5 

1 

0 

0 

0.10 

0.60 

0.40 

0 

0 

0 

0 

0 

0 

0.2284 

RPA2  32 

2 

5 

2 

0 

0 

0.10 

0.60 

0.40 

0 

0 

0 

0 

0 

0 

0.2284 

RPA2  34 

2 

6 

1 

0 

0 

0.10 

0.60 

0.40 

0 

0 

0 

0 

0 

0 

0.2284 

RPA2  35 

2 

6 

2 

0 

0 

0.10 

0.60 

0.40 

0 

0 

0 

0 

0 

0 

0.2284 

SAT3  1 

1 

1 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0.90 

0.95 

0.95 

0.2237 

RPA2  15 

1 

5 

0 

1 

1 

0 

0 

0 

0.40 

0.70 

0.55 

0.20 

0.70 

0.30 

0.1984 

RPA2  18 

1 

6 

0 

1 

1 

0 

0 

0 

0.40 

0.70 

0.55 

0.20 

0.70 

0.30 

0.1984 

RPA2  33 

2 

5 

0 

1 

1 

0 

0 

0 

0.40 

0.70 

0.55 

0.20 

0.70 

0.30 

0.1984 

RPA2  36 

2 

6 

0 

1 

1 

0 

0 

0 

0.40 

0.70 

0.55 

0.20 

0.70 

0.30 

0.1984 

SAT2  1 

1 

1 

0 

1 

0 

0 

0 

0 

0.75 

0.80 

0.95 

0 

0 

0 

0.1252 

RPA1  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.0000 

RPA2  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.0000 

RPA3  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.0000 

AC1  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.0000 

AC2  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.0000 

SAT1  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.0000 

SAT2  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.0000 

SAT3  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.0000 

In  this  example  with  broad  categorical  value  functions  and  no  capability  level  distinction 
between  the  two  bases,  many  of  the  alternatives  have  identical  scores.  However,  this  is  expected 
in  an  initial  screening  procedure  when  the  subject  matter  expert  is  tasked  to  assign  rough 
capability  level  characteristics  to  a  set  of  platforms  —  many  of  the  platfonns  will  fall  into  the 
same  value  function  category.  It  is  the  task  of  the  later  detailed  simulation  process  to  break  these 
ties  based  on  a  true  physics-based  performance  evaluation. 

Note  that  because  the  highest  value  function  categories  are  not  a  value  of  1.00,  the 
highest  possible  value  for  an  alternative  to  achieve  is  0.8177  —  when  the  alternative  scores  in  the 
"high"  category  for  all  nine  measures.  The  raw  score  for  each  alternative  can  be  translated  to  the 
percentage  of  possible  value  it  achieves,  shown  in  Table  18. 
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Table  18:  Value  achieved  for  all  alternatives 


Option 


System 

Version 

Base 

Route 

INTI 

INT2 

INT3 

Total  Score 

Percent  Value 

RPA1 

4 

1 

2 

1 

0 

0 

0.5348 

60.52% 

RPA1 

5 

1 

2 

2 

0 

0 

0.5348 

60.52% 

RPA1 

22 

2 

2 

1 

0 

0 

0.5348 

60.52% 

RPA1 

23 

2 

2 

2 

0 

0 

0.5348 

60.52% 

SAT1 

1 

1 

1 

1 

0 

0 

0.5348 

60.52% 

SAT1 

2 

1 

1 

2 

0 

0 

0.5348 

60.52% 

RPA1 

10 

1 

4 

1 

0 

0 

0.4850 

54.88% 

RPA1 

11 

1 

4 

2 

0 

0 

0.4850 

54.88% 

RPA1 

28 

2 

4 

1 

0 

0 

0.4850 

54.88% 

RPA1 

29 

2 

4 

2 

0 

0 

0.4850 

54.88% 

RPA2 

4 

1 

2 

1 

0 

0 

0.4793 

54.24% 

RPA2 

5 

1 

2 

2 

0 

0 

0.4793 

54.24% 

RPA2 

22 

2 

2 

1 

0 

0 

0.4793 

54.24% 

RPA2 

23 

2 

2 

2 

0 

0 

0.4793 

54.24% 

RPA1 

1 

1 

1 

1 

0 

0 

0.4725 

53.47% 

RPA1 

2 

1 

1 

2 

0 

0 

0.4725 

53.47% 

RPA1 

19 

2 

1 

1 

0 

0 

0.4725 

53.47% 

RPA1 

20 

2 

1 

2 

0 

0 

0.4725 

53.47% 

RPA2 

10 

1 

4 

1 

0 

0 

0.4295 

48.60% 

RPA2 

11 

1 

4 

2 

0 

0 

0.4295 

48.60% 

RPA2 

28 

2 

4 

1 

0 

0 

0.4295 

48.60% 

RPA2 

29 

2 

4 

2 

0 

0 

0.4295 

48.60% 

RPA1 

7 

1 

3 

1 

0 

0 

0.4227 

47.83% 

RPA1 

8 

1 

3 

2 

0 

0 

0.4227 

47.83% 

RPA1 

25 

2 

3 

1 

0 

0 

0.4227 

47.83% 

RPA1 

26 

2 

3 

2 

0 

0 

0.4227 

47.83% 

RPA2 

1 

1 

1 

1 

0 

0 

0.4170 

47.19% 

RPA2 

2 

1 

1 

2 

0 

0 

0.4170 

47.19% 

RPA2 

19 

2 

1 

1 

0 

0 

0.4170 

47.19% 

RPA2 

20 

2 

1 

2 

0 

0 

0.4170 

47.19% 

RPA3 

1 

1 

1 

1 

0 

0 

0.4170 

47.19% 

RPA2 

7 

1 

3 

1 

0 

0 

0.3672 

41.55% 

RPA2 

8 

1 

3 

2 

0 

0 

0.3672 

41.55% 

RPA2 

25 

2 

3 

1 

0 

0 

0.3672 

41.55% 

RPA2 

26 

2 

3 

2 

0 

0 

0.3672 

41.55% 

RPA1 

6 

1 

2 

0 

1 

1 

0.3489 

39.48% 

RPA1 

24 

2 

2 

0 

1 

1 

0.3489 

39.48% 

RPA1 

3 

1 

1 

0 

1 

1 

0.3270 

37.00% 

RPA1 

21 

2 

1 

0 

1 

1 

0.3270 

37.00% 

RPA1 

12 

1 

4 

0 

1 

1 

0.3269 

36.99% 

RPA1 

30 

2 

4 

0 

1 

1 

0.3269 

36.99% 

RPA2 

6 

1 

2 

0 

1 

1 

0.3145 

35.59% 

RPA2 

24 

2 

2 

0 

1 

1 

0.3145 

35.59% 

RPA1 

9 

1 

3 

0 

1 

1 

0.3050 

34.51% 

RPA1 

27 

2 

3 

0 

1 

1 

0.3050 

34.51% 

AC2 

1 

1 

1 

0 

1 

1 

0.2950 

33.38% 

Option 

System 

Version 

Base 

Route 

INTI 

INT2 

INT3 

Total  Score 

Percent  Value 

AC2 

2 

1 

1 

0 

2 

2 

0.2950 

33.38% 

RPA2 

3 _ 

1 

1 

0 

1 

1 

0.2926 

33.11% 

RPA2 

21 

2 

1 

0 

1 

1 

0.2926 

33.11% 

RPA2 

12 

1 

4 

0 

1 

1 

0.2925 

33.10% 

RPA2 

30 

2 

4 

0 

1 

1 

0.2925 

33.10% 

AC1 

1 

1 

1 

0 

1 

1 

0.2881 

32.60% 

AC1 

2 

1 

1 

0 

2 

2 

0.2881 

32.60% 

AC2 

3 

1 

2 

0 

1 

1 

0.2850 

32.25% 

AC2 

4 

1 

2 

0 

2 

2 

0.2850 

32.25% 

RPA1 

13 

1 

5 

1 

0 

0 

0.2839 

32.13% 

RPA1 

14 

1 

5 

2 

0 

0 

0.2839 

32.13% 

RPA1 

16 

1 

6 

1 

0 

0 

0.2839 

32.13% 

RPA1 

17 

1 

6 

2 

0 

0 

0.2839 

32.13% 

RPA1 

31 

2 

5 

1 

0 

0 

0.2839 

32.13% 

RPA1 

32 

2 

5 

2 

0 

0 

0.2839 

32.13% 

RPA1 

34 

2 

6 

1 

0 

0 

0.2839 

32.13% 

RPA1 

35 

2 

6 

2 

0 

0 

0.2839 

32.13% 

AC1 

3 

1 

2 

0 

1 

1 

0.2781 

31.47% 

AC1 

4 

1 

2 

0 

2 

2 

0.2781 

31.47% 

RPA2 

9 

1 

3 

0 

1 

1 

0.2706 

30.62% 

RPA2 

27 

2 

3 

0 

1 

1 

0.2706 

30.62% 

RPA1 

15 

1 

5 

0 

1 

1 

0.2328 

26.34% 

RPA1 

18 

1 

6 

0 

1 

1 

0.2328 

26.34% 

RPA1 

33 

2 

5 

0 

1 

1 

0.2328 

26.34% 

RPA1 

36 

2 

6 

0 

1 

1 

0.2328 

26.34% 

RPA2 

13 

1 

5 

1 

0 

0 

0.2284 

25.85% 

RPA2 

14 

1 

5 

2 

0 

0 

0.2284 

25.85% 

RPA2 

16 

1 

6 

1 

0 

0 

0.2284 

25.85% 

RPA2 

17 

1 

6 

2 

0 

0 

0.2284 

25.85% 

RPA2 

31 

2 

5 

1 

0 

0 

0.2284 

25.85% 

RPA2 

32 

2 

5 

2 

0 

0 

0.2284 

25.85% 

RPA2 

34 

2 

6 

1 

0 

0 

0.2284 

25.85% 

RPA2 

35 

2 

6 

2 

0 

0 

0.2284 

25.85% 

SAT3 

1 

1 

1 

0 

0 

1 

0.2237 

25.31% 

RPA2 

15 

1 

5 

0 

1 

1 

0.1984 

22.45% 

RPA2 

18 

1 

6 

0 

1 

1 

0.1984 

22.45% 

RPA2 

33 

2 

5 

0 

1 

1 

0.1984 

22.45% 

RPA2 

36 

2 

6 

0 

1 

1 

0.1984 

22.45% 

SAT2 

1 

1 

1 

0 

1 

0 

0.1252 

14.17% 

RPA1 

0 

0 

0 

0 

0 

0 

0.0000 

0.00% 

RPA2 

0 

0 

0 

0 

0 

0 

0.0000 

0.00% 

RPA3 

0 

0 

0 

0 

0 

0 

0.0000 

0.00% 

AC1 

0 

0 

0 

0 

0 

0 

0.0000 

0.00% 

AC  2 

0 

0 

0 

0 

0 

0 

0.0000 

c  :::: 

SAT1 

0 

0 

0 

0 

0 

0 

0.0000 

: : : : 

SAT2 

0 

0 

0 

0 

0 

0 

0.0000 

0.00% 

SAT3 

0 

0 

0 

0 

0 

0 

0.0000 

0.00% 

Performing  well  on  the  highly- weighted  INTI  sub-objective  is  the  key  to  a  high 
percentage  of  value  achievement.  However,  even  the  best  assets  achieve  only  60%  of  the 
possible  value,  so  this  screening  process  becomes  important  as  a  variety  of  assets  are  needed  to 
achieve  high  total  value  or  value  across  all  INTs. 
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Four  RPA1  platforms  and  the  two  SAT1  platforms  provide  the  highest  value  of  the 
alternatives  because  of  their  strong  INTI  capabilities.  Several  other  RPA1  and  RPA2  platfonns 
also  achieve  significant  value  through  strength  in  this  sub-objective. 

Of  the  alternatives  that  provide  INT2  and  INT3  value,  RPA1  alternatives  provide  the 
highest  six  options,  followed  by  RPA2  and  the  ACs.  When  examining  which  assets  provide 
value  to  different  INTs,  it  is  important  to  remember  that  only  one  copy  of  each  asset  is  available 
in  this  simulation.  Thus,  RPA1  could  not  be  selected  to  service  both  INTI  and  INT2  &  INT3. 

AC1  and  AC2  are  mediocre,  achieving  less  than  34%  of  the  possible  value,  and  less 
desirable  than  some  RPA1  and  RPA2  options  for  servicing  INT2  &  INT3.  SAT3,  the  only 
option  that  serves  solely  INT3,  provides  low  total  value  at  25%.  SAT2,  the  only  option  that 
serves  solely  INT2,  provides  even  lower  total  value  at  14%.  While  the  SAT2  and  SAT3  assets 
are  strong  for  their  respective  sub-objectives,  INT2  and  INT3  are  simply  weighted  too  low  for 
these  assets  to  provide  significant  overall  value.  It  is  important  to  remember  that  the  numbers 
and  values  assigned  here  are  for  illustrative  purposes  only,  as  true  values  are  known  but 
classified.  Thus,  these  results  (i.e.,  lack  of  SAT  strength)  may  be  unintuitive  and  not  match 
reality. 

These  results  help  determine  not  only  which  assets  are  vital,  but  which  platforms  of  each 
asset  are  preferred.  SAT1,  RPA1,  and  RPA2  are  the  leading  assets.  RPA1  flying  route  2  with  an 
INTI  package  provides  60.52%  of  the  total  possible  value;  SAT1  also  achieves  this  value.  The 
highest  valued  RPA2  platfonn  follows  route  2  with  an  INTI  package.  Table  19  shows  the  best 
platform  for  each  asset,  with  ties  included. 
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Table  19:  Best  platforms  for  each  asset 


Option 

System  Version 

Base 

Route 

INTI 

INT2 

INT3  Percent  Value 

RPA1  4 

1 

2 

1 

0 

0 

60.52% 

5 

1 

2 

2 

0 

0 

60.52% 

22 

2 

2 

1 

0 

0 

60.52% 

23 

2 

2 

2 

0 

0 

60.52% 

SAT1  1 

1 

1 

1 

0 

0 

60.52% 

2 

1 

1 

2 

0 

0 

60.52% 

RPA2  4 

1 

2 

1 

0 

0 

54.24% 

5 

1 

2 

2 

0 

0 

54.24% 

22 

2 

2 

1 

0 

0 

54.24% 

23 

2 

2 

2 

0 

0 

54.24% 

RPA3  1 

1 

1 

1 

0 

0 

47.19% 

AC2  1 

1 

1 

0 

1 

1 

33.38% 

2 

1 

1 

0 

2 

2 

33.38% 

AC1  1 

1 

1 

0 

1 

1 

32.60% 

2 

1 

1 

0 

2 

2 

32.60% 

SAT3  1 

1 

1 

0 

0 

1 

25.31% 

SAT2  1 

1 

1 

0 

1 

0 

14.17% 

When  beginning  to  create  a  portfolio  of  ISR  assets,  it  would  be  advantageous  to  start  with 
alternatives  from  this  set. 

The  methodology  developed  allows  for  two  main  screening  processes.  First,  the  assets 
under  consideration  can  be  explored  for  simple  value  dominance.  If  every  platfonn  of  Asset  A 
provides  less  value  than  every  platform  of  Asset  B,  Asset  A  can  be  eliminated  under  a  rough  of 
screening  procedure.  This  screening  applies  when  the  goal  is  achieving  overall  value  and  when 
constraints  prevent  the  selection  of  both  assets.  For  example,  if  within  a  larger  problem  only  one 
SAT  asset  can  be  selected,  both  SAT1  platforms  (60.52%)  dominate  SAT3  (25%)  and  SAT2 
(14%),  so  SAT2  and  SAT3  can  be  removed  from  the  pool  of  possible  alternative.  It  must  be 
noted  that  SAT2  and  SAT3  provide  different  INT  capabilities  than  SAT1,  and  removing  them 
from  the  alternative  pool  could  be  detrimental  to  an  ideal  solution. 

However,  if  a  constraint  states  firmly  that  only  one  SAT  asset  can  be  used,  SAT  1  will 
provide  the  most  value  to  the  overall  hierarchy.  Removing  just  two  assets  (and  only  two 
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alternatives,  as  each  asset  has  only  one  platfonn)  still  results  in  a  large  reduction  in  the  number 
of  portfolios  for  evaluation  and  thus  the  time  required  for  evaluation. 


Table  20:  Alternative  and  combination  reduction 


Original 

After  Screening 

%  Reduction 

Total  Alternatives 

93 

91 

2.15% 

Total  Combinations 

821400 

205350 

75.00% 

As  shown  in  Table  20,  a  two  percent  reduction  in  alternatives  results  in  a  75%  reduction 
in  combinations. 

This  simple  value  dominance  can  dangerously  eliminate  versatile  alternatives. 

Narrowing  dominance  to  the  sub-objective  level,  when  value  in  a  certain  sub-objective  is  desired 
in  addition  to  overall  value,  can  help  mitigate  this  danger.  A  portfolio  goal  may  be  to  achieve 
value  in  a  set  of  INT  sub-objectives  in  addition  to  maximizing  overall  value.  The  capability 
level  assessment  process  allows  for  an  alternative's  total  value  achieved  to  be  broken  down  into 
value  achieved  in  each  sub-objective. 

If  every  alternative  for  Asset  A  provides  more  value  than  every  alternative  for  Asset  B 
for  a  specific  set  of  sub-objectives  of  interest,  then  Asset  B  is  thoroughly  dominated.  Depending 
on  the  constraints  of  the  problem,  Asset  B  can  be  eliminated  as  it  will  not  be  chosen  to  provide 
value  in  the  set  of  sub-objectives  over  Asset  A. 

A  single  sub-objective  example  involves  SAT1,  RPA2  versions  that  provide  INTI 
capability,  and  RPA3.  Each  of  these  assets  provides  INTI  capability.  The  value  generated  from 
the  capability  level  of  SAT  1  dominates  the  value  generated  from  all  versions  of  RPA2  and  RPA3 
in  this  sub-objective  category.  If  only  one  alternative  can  be  chosen  to  provide  INTI  capability, 
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SAT1  would  be  chosen  and  the  INTI  versions  of  RPA2  and  RPA3  would  be  screened, 
eliminating  a  large  number  of  alternatives. 

A  multiple  sub-objective  example  involves  the  platforms  of  RPA1,  RPA2,  AC1,  and  AC2 
that  provide  INT2  and  INT3  capabilities.  While  RPA1  and  RPA2  platfonns  score  highest  in 
these  sub-objectives,  several  RPA1  and  RPA2  platforms  also  rank  below  AC1  and  AC2.  Thus, 
in  this  situation,  none  of  the  assets  could  be  screened  on  total  dominance.  However,  if  there 
existed  a  third  "AC3"  asset  that  provided  a  value  score  of  <  22.45%  in  INT2  and  INT3  for  all 
AC3  platfonns,  it  would  be  dominated  by  the  existing  assets  in  those  sub-objectives  and  could  be 
eliminated. 

The  method  just  described  involves  screening  an  asset  based  on  total  dominance  by 
another  asset  —  when  all  platforms  of  one  asset  dominate  all  platforms  of  another  asset  in  a  set  of 
sub-objectives.  In  reality,  this  situation  may  be  rare.  The  second  screening  method  is  based  on 
alternative  dominance  within  a  specific  asset.  RPA1  has  37  possible  platforms  as  a  result  of  the 
combinations  of  its  basing  options,  routes,  and  INT  packages,  but  only  one  can  be  selected  for 
deployment.  Instead  of  including  all  37  alternatives  in  the  portfolio  modeling  process,  it  may  be 
valuable  to  examine  only  a  select  few  top  scoring  RPA1  platforms.  The  top  scoring  alternatives 
within  an  asset  are  selected  for  further  analysis,  while  the  remainder  are  screened  out.  For  an 
asset  such  as  RPA1  that  has  two  sub-objective  capability  coverage  options  (INTI  or  INT2  & 
INT3),  it  is  important  to  screen  widely  enough  to  include  RPA1  platforms  that  cover  both  areas. 
This  way,  constraints  dealing  with  sub-objective  coverage  or  limitations  in  number  of  assets  can 
be  met  without  overly-limiting  the  possible  solutions.  Thus,  one  would  select  RPA1  versions  4, 

5,  22,  and  23,  which  score  highest  (60.52%)  and  provide  INTI  capability,  as  well  as  RPA1 
versions  6  and  24,  which  score  highest  (39.48%)  for  the  INT2  &  INT3  package.  When  ties 


79 


occur,  all  alternatives  are  included.  The  performance  is  not  truly  identical,  but  appears  this  way 


due  to  the  categorical  value  functions.  More  detailed  modeling  will  resolve  these  ties  and 


determine  the  true  best  alternative.  Following  this  model,  the  alternative  set  is  reduced  to  the  21 


alternatives  in  Table  21. 


Table  21:  Alternative  reduction  based  on  value  in  sub-objective  areas 

Option 


System 

Version 

Base 

Route 

INTI 

INT2 

INT3 

Percent  Value 

RPA1 

4 

1 

2 

1 

0 

0 

60.52% 

5 

1 

2 

2 

0 

0 

60.52% 

22 

2 

2 

1 

0 

0 

60.52% 

23 

2 

2 

2 

0 

0 

60.52% 

6 

1 

2 

0 

1 

1 

39.48% 

24 

2 

2 

0 

1 

1 

39.48% 

RPA2 

4 

1 

2 

1 

0 

0 

54.24% 

5 

1 

2 

2 

0 

0 

54.24% 

22 

2 

2 

1 

0 

0 

54.24% 

23 

2 

2 

2 

0 

0 

54.24% 

6 

1 

2 

0 

1 

1 

35.59% 

24 

2 

2 

0 

1 

1 

35.59% 

RPA3 

1 

1 

1 

1 

0 

0 

47.19% 

AC1 

1 

1 

1 

0 

1 

1 

32.60% 

2 

1 

1 

0 

2 

2 

32.60% 

AC2 

1 

1 

1 

0 

1 

1 

33.38% 

2 

1 

1 

0 

2 

2 

33.38% 

SAT1 

1 

1 

1 

1 

0 

0 

60.52% 

2 

1 

1 

2 

0 

0 

60.52% 

SAT2 

1 

1 

1 

0 

1 

0 

14.17% 

SAT3 

1 

1 

1 

0 

0 

1 

25.31% 

This  reduction  keeps  the  highest-scoring  alternatives  and  still  allows  for  a  robust  portfolio 
selection  in  the  face  of  enforced  limitations.  Many  possibilities  exist  to  provide  high  overall 
value  and  achieve  value  in  the  three  INT  sub-objectives,  no  matter  asset  type  or  total  number  of 
assets  constraints. 
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Table  22:  Alternative  and  combination  reduction 


Original 

After  Screening 

%  Reduction 

Total  Alternatives 

93 

29 

68.82% 

Total  Combinations 

821400 

10584 

98.71% 

Time  to  solve  (days) 

4278.1 

55.1 

98.71% 

As  shown  in  Table  22,  this  procedure  reduces  the  total  alternatives  by  68.82%  and 
reduces  the  number  of  possible  portfolios  by  98.71%.  Originally,  the  time  to  evaluate  all 
portfolios  was  4,278  days,  or  1 1.72  years  —  an  impossible  amount  of  time.  Now,  the  time  to 
evaluate  all  possible  portfolios  is  reduced  to  55. 1  days.  Not  ideal,  but  a  possible  evaluation  time. 
This  was  a  conservative  screening,  as  many  alternatives  with  inferior  total  value  remained.  More 
aggressive  screening  would  further  reduce  the  time  for  evaluation. 

This  process  establishes  a  set  of  alternatives  to  serve  as  a  starting  point  for  a  response 
surface  search  algorithm  that  utilizes  a  physics-based  simulation  program  for  true  portfolio 
performance  evaluation.  Even  if  no  alternatives  are  truly  eliminated  from  the  search,  by 
providing  the  algorithm  with  a  good  starting  point  based  on  the  set  of  alternatives  found  above, 
the  total  search  time  is  likely  reduced.  Actually  eliminating  inferior  alternatives  speeds  the 
search  even  more. 

Sufficiency  Screening. 

After  screening  inferior  assets  or  alternatives  to  reduce  the  number  of  portfolios  to  a  more 
reasonable  size,  a  complex  simulation  program  is  implemented  to  accurately  assess  portfolio 
perfonnances.  An  algorithm  searches  for  the  highest-performing  portfolio,  effectively  ranking 
portfolios  as  it  progresses. 

While  determining  the  highest-performing  portfolio  is  valuable,  this  may  not  be  the 
actual  goal.  Often,  it  is  more  valuable  to  detennine  which  portfolios  are  sufficient  to  meet  the 
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needs  of  the  problem,  and  which  are  not.  This  becomes  important  when  cost  is  considered  in  the 
implementation  of  a  portfolio. 

In  the  ISR  example,  the  best  perfonning  portfolio  will  simply  use  all  available  assets  in 
their  ideal  configurations  —  a  trivial  solution.  This  best  portfolio  may  also  be  the  most 
expensive,  yet  provide  only  marginal  additional  value  over  another  portfolio  that  still  sufficiently 
solves  the  problem  but  costs  less.  It  is  important  to  identify  which  ISR  asset  portfolios  will  meet 
the  needs;  any  portfolio  perfonning  above  the  requirement  can  still  be  selected,  though  extra 
resources  potentially  required  may  be  allocated  better  elsewhere. 

A  process  is  needed  to  define  this  sufficiency  level,  so  the  portfolio  set  can  be  divided 
into  acceptable  and  unacceptable  sets.  From  there,  post-analysis  can  be  perfonned  on  the 
acceptable  portfolios  based  on  cost  and  other  factors.  Fundamentally,  portfolio  sufficiency  is 
derived  from  the  opinions  of  the  decision  maker.  As  previously  discussed,  for  many  problems  a 
rough  minimum  acceptable  performance  level  of  a  solution  is  known.  The  difficulty  is 
detennining  which  portfolios  achieve  that  performance  given  the  inherent  uncertainties  in  both 
assigning  the  level  and  determining  true  portfolio  performance. 

The  sufficiency  detennination  process  for  the  ISR  problem  begins  at  the  analyst  level. 

The  raw  data  collected  from  ISR  assets  is  transformed  into  actionable  intelligence  by  a  set  of 
analysts.  An  analyst  who  works  with  this  data  on  a  regular  basis  first  creates  a  value  function  to 
map  the  characteristics  of  the  data  he  receives  to  the  value  of  useable  information  generated. 
Depending  on  the  analyst's  specialty  area,  this  characteristic  could  be  quantity  of  data,  quality  of 
the  data,  timeliness  of  the  data,  or  some  combination.  The  analyst  uses  past  experiences  and 
future  expectations  to  estimate  the  value  of  infonnation  that  can  be  created  from  a  certain  raw 
data  input.  For  example,  an  imagery  analyst  may  feel  confident  that  he  can  provide  a  high  level 
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of  intelligence  (i.e.,  90%  of  the  maximum  value  he  could  possibly  provide)  with  10  photographs 
a  day.  At  three  photographs  a  day,  the  value  he  can  provide  drops  to  40%.  Thus,  a  value 
function  is  created.  Depending  on  the  resolution  desired,  "analyst"  could  mean  an  actual 
individual  person,  or  a  larger  analysis  shop  that  focuses  in  a  specific  area. 

After  value  functions  are  created  by  each  analyst,  the  top-level  decision  maker  is  asked  to 
estimate  the  value  he  wishes  a  successful  portfolio  to  provide  on  a  zero-to-one  scale,  with  one 
being  the  absolute  highest  value  imaginable.  This  definition  depends  on  the  specific  problem 
under  consideration.  If  the  decision  maker  is  determining  an  appropriate  portfolio  for  the 
reallocation  of  a  portion  of  current  assets  to  a  new  mission  area,  he  would  provide  a  number  that 
represents  the  percentage  of  overall  value  the  new  mission  is  worth,  or  the  percentage  of  total 
infonnation  gathering  capability  he  wishes  to  allocate  to  the  new  mission. 

In  a  complete  force  structuring  problem,  such  as  the  ISR  problem  under  consideration, 
the  goal  is  to  develop  a  portfolio  of  assets  that  that  efficiently  and  sufficiently  allocates  data  to  all 
analysts.  In  this  way,  no  analyst  would  be  flooded  by  and  none  would  be  starved  for  raw  data, 
and  all  would  have  the  resources  available  to  perfonn  their  job.  The  decision  maker  would 
understand  the  inefficient  status  quo  as  well  as  the  value  function  process  that  analysts 
performed.  The  decision  maker  then  provides  a  number  that  represents  the  minimum  level  of 
value  he  deems  necessary  from  an  acceptable  portfolio.  Clearly,  the  closer  this  number  is  to  one, 
the  higher  the  perfonnance  needed  for  a  sufficient  portfolio.  If  the  value  of  the  current 
operational  portfolio  is  known,  the  decision  maker  may  wish  for  a  percentage  improvement  on 
current  capabilities. 

Once  the  decision  maker  has  provided  a  desired  value,  this  number  is  transferred  back 
down  to  the  analyst  level.  This  value  is  found  on  the  v-axis  of  each  analyst's  value  function 
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graph,  and  the  value  function  itself  maps  this  value  level  to  a  certain  level  of  data  characteristic 


on  the  x-axis,  as  shown  in  Figure  17. 


Figure  17:  Analyst  value  function 


Now,  the  decision  maker's  single  input  has  been  pushed  throughout  to  detennine  the  minimum 
data  each  analyst  should  receive.  While  the  analysts  have  completely  different  raw  data 
requirements,  each  will  be  provided  whatever  amount  necessary  to  achieve  the  same  value  that 
results  from  processing  that  data.  This  avoids  the  disparity  in  data  provision  that  prevents 
optimal  collective  intelligence  generation. 

The  sufficiency  level  is  set  at  the  ability  to  provide  this  level  of  data.  A  portfolio  can  be 
evaluated  on  whether  its  performance  results  in  sufficient  data  to  meet  the  need  of  each  analyst. 
If  a  portfolio  can  meet  this  criterion,  it  is  deemed  sufficient  and  set  aside  for  further  post¬ 
analysis.  If  a  portfolio  cannot  meet  this  data  criterion,  it  is  deemed  insufficient  and  screened 
from  further  study.  This  screening  procedure  based  on  data  sufficiency  is  thus  used  to  further 
reduce  the  number  of  portfolios  that  undergo  final  analysis  to  only  those  that  sufficiently  solve 
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the  problem.  In  the  post-analysis  phase,  factors  such  as  cost  determine  the  selection  of  an  ideal 
portfolio. 

IV.D  Summary 

The  purpose  of  this  chapter  was  to  provide  an  application  of  several  novel  screening 
techniques  to  an  actual  USSTRATCOM  JFCC-ISR  hierarchy  with  a  sample  alternative  data  set. 
The  methodology  developed  here  transforms  a  broad  perfonnance-based  hierarchy  designed  for 
portfolio  evaluation  to  a  simple  capability-based  hierarchy  designed  for  alternative  evaluation. 
Eliminating  a  number  of  alternatives  results  in  significantly  fewer  portfolios,  allowing  sufficient 
portfolios  to  be  evaluated  against  the  perfonnance-based  hierarchy  and  examined  on  a  reasonable 
timescale. 
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Conclusions  and  Recommendations 


V.A.  Introduction 

The  continuous  advancement  of  technology  has  provided  incredible  opportunities  for  the 
Department  of  Defense  to  achieve  its  mission.  Innovation  in  hardware  and  software  has  allowed 
for  an  explosion  in  the  number  of  intelligence,  surveillance,  and  reconnaissance  platforms 
available.  This  progress  has  coincided  with  the  wars  in  Iraq  and  Afghanistan,  allowing  United 
States  efforts  to  be  aided  by  more,  better  data. 

A  by-product  of  new  ISR  technology  becoming  available  immediately  when  needed  has 
been  the  inefficient  deployment  of  such  assets.  Without  proper  asset  distribution  based  on 
platform  performance  and  user  needs,  capability  is  wasted.  Some  analysis  units  are  flooded  with 
data  impossible  to  process  in  a  timely  manner,  while  others  starve  for  data  that  could  provide 
actionable  intelligence. 

The  establishment  by  USSTRATCOM  JFCC-ISR  of  an  intelligence  objectives  hierarchy 
is  a  good  start  to  improving  asset  deployment.  Next,  it  must  be  detennined  what  minimum  level 
of  data  is  necessary  in  which  analysis  areas  so  that  an  appropriate  portfolio  of  ISR  assets  can  be 
generated  to  sufficiently  meet  the  objectives.  Difficulty  arises  as  there  are  hundreds  of  ISR 
platform  variations  leading  to  an  astronomical  number  of  possible  portfolios.  This  research 
developed  methodologies  to  confront  both  of  these  challenges.  A  process  was  explored  to 
eliminate  poor  ISR  alternatives  and  thus  reduce  the  number  of  portfolios,  followed  by  a 
procedure  to  define  portfolio  sufficiency. 
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V.B.  Research  Contributions 


The  purpose  of  this  research  was  to  contribute  to  the  field  of  Decision  Analysis  a  new 
methodology  to  examine  multiobjective  decision  problems  with  combinatorial  alternative  sets. 
Specifically,  the  procedures  developed  apply  when  each  objective  is  evaluated  through  the  same 
set  of  sub-objectives,  a  common  decision  problem  characteristic.  This  research  established 
processes  to  reduce  the  size  of  the  alternative  set  through  intelligent  screening  and  to  determine 
the  sufficiency  of  portfolios  of  alternatives. 

The  first  contribution  was  in  the  area  of  alternative  screening.  Many  problems  feature  a 
large  number  of  alternatives,  especially  as  each  variation  of  a  general  solution  can  be  considered 
a  new  alternative.  When  the  decision  maker  may  select  a  portfolio  of  these  alternatives  to 
generate  a  solution,  the  number  of  possibilities  becomes  enormous;  evaluating  all  portfolios  is 
difficult. 

Eliminating  a  small  number  of  alternatives  can  drastically  reduce  the  number  of  possible 
portfolios.  A  procedure  was  developed  to  reduce  the  multi-objective  perfonnance-based 
hierarchy  to  a  simple  capability-based  hierarchy.  This  allowed  for  assessment  of  alternative 
contribution  to  the  overall  goal  through  both  basic  capability  and  slightly  more  advanced 
estimated  performance.  With  these  assessments  made,  low-perfonning  alternatives  may  be 
eliminated  depending  on  the  constraints  of  the  specific  problem  under  evaluation.  This 
procedure  commonly  reduces  the  number  of  possible  portfolios  by  greater  than  95%. 

The  second  contribution  to  the  field  of  decision  analysis  was  in  the  area  of  alternative 
sufficiency.  It  is  valuable  to  sort  alternatives,  or  portfolios  of  alternatives,  into  two  groups:  those 
that  successfully  meet  the  minimum  requirements  of  the  problem  and  those  that  fail  to  do  so. 
Then,  the  failed  alternatives  can  be  removed  from  further  consideration. 
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In  problems  where  the  decision  maker  is  distant  from  the  lowest  tier  in  the  evaluative 
hierarchy,  the  performance  valued  by  the  decision  maker  is  difficult  to  translate  into  the  bottom 
tier  measure  requirements  an  alternative  must  provide.  Thus,  a  method  was  developed  to  push 
the  decision  maker's  needs  down  the  hierarchy.  This  process  combined  perfonnance  or 
confidence  requirements  from  the  decision  maker  with  the  knowledge  of  those  familiar  with  the 
specifics  of  the  bottom  tier  of  the  hierarchy,  thereby  determining  the  level  of  measure 
perfonnance  necessary  to  sufficiently  achieve  the  goal.  A  portfolio  of  alternatives  could  then  be 
rated  on  whether  it  provided  an  acceptable  level  of  performance  in  each  measure  and  thus  sub¬ 
objective. 

V.C.  Recommendations  for  Further  Research 

This  research  developed  a  methodology  that  was  shown  to  successfully  achieve  its 
mission  of  eliminating  alternatives.  A  theoretical  set  of  alternatives  was  analyzed  and  screened 
according  to  various  constraints,  demonstrating  the  ability  of  the  procedure  to  generate  a  large 
percentage  reduction  of  the  portfolio  space. 

Future  research  should  attempt  to  validate  this  methodology  through  application  to  a  full 
problem  where  the  performance  of  a  large  number  of  portfolios  is  known.  The  procedure  can  be 
followed  to  eliminate  alternatives;  according  to  the  theory  presented  in  this  research,  the  best¬ 
performing  portfolios  should  be  comprised  of  alternatives  that  remain  after  screening. 

These  validation  studies  are  necessary  to  determine  the  rigor  of  the  methodology  and 
identify  potential  process  improvements.  Additional  research  can  focus  on  refining  the 
methodology  and  expanding  the  universe  of  problems  to  which  it  can  be  applied,  or  adapting  it 
for  a  new  type  of  problem.  Finally,  an  expansion  of  the  portfolio  sufficiency  process  described 


here  can  be  performed.  Areas  of  investigation  could  include  sensitivity  analysis,  sub-objective 
perfonnance  to  objective  achievement  mapping,  and  minimum  sub-objective  achievement  issues. 

V.D.  Conclusions 

In  a  world  growing  more  interconnected  and  data  driven,  decision  makers  face  problems 
more  complex  than  ever  before.  Failing  to  give  a  problem  the  careful  study  it  deserves  can  lead 
to  a  poor  decision  with  potentially  disastrous  results.  At  the  very  least,  the  decision  maker  may 
fail  to  obtain  all  the  value  possible. 

By  reducing  the  complexity  of  the  hierarchy,  the  analyst  can  reduce  the  size  of  the 
alternative  set  and  number  of  potential  portfolios.  Eliminating  poor  alternatives  allows  for  better 
concentration  on  the  remaining  solutions,  improving  the  chances  that  a  good  decision  will  be 
made. 

The  new  methodology  developed  in  this  research  for  screening  alternatives  and 
detennining  sufficiency  criteria  was  applied  to  the  USSTRATCOM  hierarchy  with  a  sample 
alternative  set.  The  results  demonstrated  that  a  capability-based  assessment  can  greatly  reduce 
the  number  of  alternatives,  making  perfonnance  evaluation  of  the  portfolios  comprised  of  good 
alternatives  more  feasible.  Applying  this  process  to  the  full  ISR  force  sizing  problem  will  allow 
for  an  ideal  mix  of  assets  to  be  identified.  This  portfolio  will  provide  the  data  needed  to  generate 
quality  intelligence  to  the  benefit  of  the  military  and  the  United  States. 
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Appendix  B. 


Blue  Dart 

Researchers  at  the  Air  Force  Institute  of  Technology  (AFIT)  are  leading  the  way  on 
ground  breaking  research  to  help  commanders  obtain  the  intelligence,  surveillance,  and 
reconnaissance  (ISR)  assets  needed  to  fight  and  win  wars.  The  recent  United  States  military 
efforts  in  Iraq  and  Afghanistan  have  spurred  the  development  of  innovative  surveillance  and 
reconnaissance  technology  to  meet  increasing  data  requirements  and  generate  actionable 
intelligence.  In  today's  complex  ISR  world,  there  are  billions  of  combinations  of  platfonns, 
sensors,  and  concepts  of  operations  possible  to  help  a  commander  understand  the  ever-changing 
battlefield.  The  time  to  model  and  evaluate  the  effectiveness  of  these  combinations,  even  with 
the  latest  advancements  in  computing  technology,  is  prohibitive,  on  the  order  of  many  years. 

This  is  unacceptable  given  today's  immediate  and  crucial  operations. 

AFIT  researchers  are  engaged  in  advanced  study  of  ideal  ISR  asset  distribution  to 
optimize  intelligence  gathering.  New  assets  have  been  rapidly,  but  inefficiently,  deployed;  poor 
coordination  across  the  Department  of  Defense  has  led  to  a  failure  to  meet  the  needs  of  the  many 
units  that  require  appropriate  information  on  a  timely  basis.  Some  organizations  drown  in  data 
as  numerous  assets  stream  terabytes  of  data  to  be  processed.  Meanwhile,  other  groups  suffer 
from  a  lack  of  the  data  needed  to  develop  actionable  intelligence.  AFIT  is  developing 
sophisticated  mathematical  models  to  ensure  an  ideal  portfolio  of  ISR  assets  is  deployed  to 
successfully  meet  the  complexity  of  today's  joint  environment. 

Difficulty  arises  when  attempting  to  identify  the  ideal  portfolio.  The  Department  of 
Defense  possesses  many  different  types  of  assets  to  provide  ISR  capability.  For  each  asset, 
multiple  equipment  packages  may  be  installed  to  generate  different  types  of  data.  Assets  can  be 
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used  in  a  number  of  locations  in  a  variety  of  ways.  These  variables  and  others  lead  to  many 
possible  usage  alternatives  for  a  single  asset.  In  the  context  of  Department  of  Defense  total 
assets  and  theatre  deployment  options,  an  enonnous  number  of  possible  portfolios  are  generated 
through  the  combination  of  different  asset  alternatives.  Computer  modeling  of  every  portfolio's 
perfonnance  requires  an  impossibly  long  time  on  even  the  fastest  of  computer  systems.  A 
process  is  needed  to  select  more  efficiently  an  ideal  portfolio. 

AFIT  has  invented  a  two-part  methodology  for  a  class  of  portfolio  selection  problems 
such  as  the  ISR  situation  previously  described,  when  there  exist  multiple  objectives  that  are  all 
judged  on  the  same  hierarchical  metric  set.  First,  a  novel  capability-based  screening  process  was 
developed  to  evaluate  ISR  asset  alternatives  on  intended  capability  through  the  reduction  of  the 
multiobjective  performance  hierarchy.  Poor  alternatives  are  eliminated,  reducing  the  size  of  the 
alternative  set.  Removing  only  a  few  alternatives  greatly  reduces  the  number  of  possible 
portfolios,  allowing  for  better  examination  of  the  remaining  portfolios. 

Next,  a  method  was  developed  to  define  portfolio  sufficiency  according  to  the 
requirements  of  the  decision  maker.  The  portfolios  remaining  after  the  capability-based 
screening  are  then  evaluated  on  a  performance  basis  to  determine  sufficiency  or  not.  With  a  set 
of  sufficient  portfolios  comprised  of  well-performing  alternatives,  higher  resolution  post-analysis 
methods  can  be  applied  to  choose  a  solution. 

This  two-part  procedure  was  applied  to  an  ISR  objective  hierarchy  developed  by  the 
United  States  Strategic  Command  Joint  Functional  Component  Command  for  Intelligence, 
Surveillance  and  Reconnaissance.  After  deconstructing  the  hierarchy,  a  set  of  representative 
alternatives  were  evaluated  and  a  variety  of  screening  procedures  were  applied  to  demonstrate 
the  significant  reduction  in  the  number  of  portfolios  possible.  In  one  such  screening  process, 


98 


researchers  were  able  to  reduce  the  amount  of  time  needed  to  examine  good  force  mixes  from  1 1 
years  to  two  months.  This  process  allows  for  more  resources  to  be  dedicated  to  estimating  the 
true  perfonnance  of  the  sufficient  portfolios;  new  systems  can  still  be  fielded  rapidly,  but  now 
optimized  to  help  the  soldiers,  sailors,  ainnen  and  marines  in  the  field. 

The  procedure  developed  at  AF1T  and  applied  to  1SR  distribution  can  be  generalized  to 
any  complex  multiobjective  problem  with  the  hierarchical  structure  described  in  which  a 
portfolio  of  alternatives  is  permitted.  Many  military  force  structuring  problems  use  this  type  of 
hierarchical  decision  making  structure.  Eliminating  poor  alternatives  initially  through  an 
intelligent  screening  process,  as  well  as  only  focusing  on  sufficient  portfolios,  can  greatly  reduce 
the  number  of  portfolios  under  consideration.  This  smaller  universe  of  solutions  increases  the 
likelihood  the  ideal  or  a  near-ideal  solution  will  be  selected.  In  an  environment  with  limited 
resources,  the  work  of  AFIT’s  researchers  to  optimize  assets  may  prove  the  difference  between 
failure  and  success. 

The  views  expressed  in  this  article  are  those  of  the  author  and  do  not  reflect  the  official  policy  or 
position  of  the  United  States  Air  Force,  Department  of  Defense,  or  the  US  Government. 
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